  | 
 IDS Forum
 Re: wait for mutex
 Posted By: Madison Pruet Date: Wednesday, 20 August 2014, at 11:05 a.m.
In Response To: Re: wait for mutex (FRANK)
  
In 12.10xc4, we rewrote the way that the ER queues were locked. This
 significantly reduced lock contention and improved ER performance by close
 to 1200%
 Sent from my iPad
 > On Aug 20, 2014, at 8:49 AM, "FRANK" <yunyaoqu@gmail.com> wrote:
 >
 > Thanks, Fernando!
 >
 > Here it is, confused ...
 > We indeed have ER slow/ data queued issue now
 >
 > Frank
 >
 > informix@walter ~]$ onstat -g wmx
 > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:05
 > -- 11847728 Kbytes
 > Mutexes with waiters:
 > mid addr name holder lkcnt waiter
 > waittime
 > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 227 0
 >
 > 5504 0
 >
 > 98 0
 >
 > 196 0
 >
 > 107 0
 >
 > 5488 0
 > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 17
 >
 > 4430 17
 >
 > 4585 17
 >
 > 4754 17
 >
 > 4528 17
 >
 > 4481 17
 >
 > 4897 17
 >
 > 4527 17
 >
 > 4526 17
 >
 > 4608 17
 >
 > 4752 17
 >
 > 214 17
 >
 > 205 17
 >
 > 195 16
 >
 > 4554 16
 >
 > 206 16
 >
 > 213 16
 >
 > 4618 16
 >
 > 4695 16
 >
 > 4425 16
 >
 > 4619 16
 >
 > 4560 16
 >
 > 4428 16
 >
 > 4500 16
 >
 > 4448 16
 >
 > 4555 16
 >
 > 4603 16
 >
 > 4609 16
 >
 > 4584 16
 >
 > 4607 16
 >
 > 4502 16
 >
 > 4447 16
 >
 > 4604 16
 >
 > 4896 16
 >
 > 4602 16
 >
 > 4895 16
 >
 > 4753 16
 >
 > 204 16
 >
 > 4426 16
 >
 > 4634 16
 > Number of mutexes on VP free lists: 10735
 > Read/Write Locks
 > addr id name slocks xlock waiters
 > [informix@walter ~]$ onstat -g lmx
 > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:17
 > -- 11847728 Kbytes
 > Locked mutexes:
 > mid addr name holder lkcnt waiter
 > waittime
 > 6307 1d9c113d0 qtrMutex31 5488 0
 > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 5495 0
 >
 > 5488 0
 > 6547 1d9fbcb90 rqrest_trg_send 5496 0
 > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 29
 >
 > 4430 29
 >
 > 4585 29
 >
 > 4754 29
 >
 > 4528 29
 >
 > 4481 29
 >
 > 4897 29
 >
 > 4527 29
 >
 > 4526 29
 >
 > 4608 29
 >
 > 4752 29
 >
 > 214 29
 >
 > 205 29
 >
 > 195 28
 >
 > 4554 28
 >
 > 206 28
 >
 > 213 28
 >
 > 4618 28
 >
 > 4695 28
 >
 > 4425 28
 >
 > 4619 28
 >
 > 4560 28
 >
 > 4428 28
 >
 > 4500 28
 >
 > 4448 28
 >
 > 4555 28
 >
 > 4603 28
 >
 > 4609 28
 >
 > 4584 28
 >
 > 4607 28
 >
 > 4502 28
 >
 > 4447 28
 >
 > 4604 28
 >
 > 4896 28
 >
 > 4602 28
 >
 > 4895 28
 >
 > 4753 28
 >
 > 204 28
 >
 > 4426 28
 >
 > 4634 28
 > Number of mutexes on VP free lists: 12879
 > Read/Write Locks
 > addr id name slocks xlock waiters
 > [informix@walter ~]$
 >
 > On Wed, Aug 20, 2014 at 10:37 AM, Fernando Nunes <domusonline@gmail.com>
 > wrote:
 >
 > > onstat -g wmx and onstat -g lmx to see what is the mutex...
 > > It can be hard to "see"...
 > > Regards
 > >
 > > On Wed, Aug 20, 2014 at 3:34 PM, FRANK <yunyaoqu@gmail.com> wrote:
 > >
 > > > Folks,
 > > >
 > > > Normally I do not see S---- in output of onstat -u, but now I see.
 > > >
 > > > Any comments?
 > > >
 > > > Thanks
 > > > Frank
 > > >
 > > > RHEL 6
 > > >
 > > > [informix@walter ngdc_er_20140820]$ onstat -u
 > > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up
 00:38:50
 > > > -- 11847728 Kbytes
 > > > Userthreads
 > > > address flags sessid user tty wait tout
 > > > locks nreads nwrites
 > > > 1d81b3028 ---P--D 1 informix - 0 0
 > > > 0 398 1015
 > > > 1d81b3880 ---P--F 0 informix - 0 0
 > > > 0 0 12778
 > > > 1d81b40d8 ---P--F 0 informix - 0 0
 > > > 0 0 18875
 > > > 1d81b4930 ---P--F 0 informix - 0 0
 > > > 0 0 11092
 > > > 1d81b5188 ---P--F 0 informix - 0 0
 > > > 0 0 18161
 > > > 1d81b59e0 ---P--F 0 informix - 0 0
 > > > 0 0 26058
 > > > 1d81b6238 ---P--F 0 informix - 0 0
 > > > 0 0 35724
 > > > 1d81b6a90 ---P--F 0 informix - 0 0
 > > > 0 0 12501
 > > > 1d81b72e8 ---P--F 0 informix - 0 0
 > > > 0 0 36521
 > > > 1d81b7b40 ---P--F 0 informix - 0 0
 > > > 0 0 15588
 > > > 1d81b8398 ---P--F 0 informix - 0 0
 > > > 0 0 46021
 > > > 1d81b8bf0 ---P--F 0 informix - 0 0
 > > > 0 0 2964
 > > > 1d81b9448 ---P--F 0 informix - 0 0
 > > > 0 0 183203
 > > > 1d81b9ca0 ---P--F 0 informix - 0 0
 > > > 0 0 47100
 > > > 1d81ba4f8 ---P--F 0 informix - 0 0
 > > > 0 0 18092
 > > > 1d81bad50 ---P--F 0 informix - 0 0
 > > > 0 0 47221
 > > > 1d81bb5a8 ---P--F 0 informix - 0 0
 > > > 0 0 13200
 > > > 1d81bbe00 ---P--- 18 informix - 0 0
 > > > 0 0 63
 > > > 1d81bc658 ---P--B 19 informix - 0 0
 > > > 0 515817 4096
 > > > 1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
 > > > 0 459361 3977
 > > > 1d81bdf60 ---P--D 24 informix - 0 0
 > > > 0 4 0
 > > > 1d81be7b8 ---P--D 23 informix - 0 0
 > > > 0 0 0
 > > > 1d81bf010 ---P--- 43 informix - 0 0
 > > > 1 240 21860
 > > > 1d81bf868 Y--P--- 143 informix - 1db2aadc8 0
 > > > 1 8 22528
 > > > 1d81c00c0 ---P--- 33 informix - 0 0
 > > > 0 0 0
 > > > 1d81c0918 Y--P--- 141 informix - 1db2ccb78 0
 > > > 1 9 3
 > > > 1d81c1170 ---P--- 35 informix - 0 0
 > > > 1 0 0
 > > > 1d81c19c8 Y--P--- 36 informix - 1d9910448 0
 > > > 3 0 0
 > > > 1d81c2220 Y--P--D 41 informix - 48bdf970 0
 > > > 0 0 0
 > > > 1d81c2a78 ---P--- 44 informix - 0 0
 > > > 1 666 32901
 > > > 1d81c32d0 ---P--- 45 informix - 0 0
 > > > 1 711 32575
 > > > 1d81c3b28 ---P--- 46 informix - 0 0
 > > > 0 3 0
 > > > 1d81c4380 ---P--- 47 informix - 0 0
 > > > 1 240 442
 > > > 1d81c4bd8 ---P--- 48 informix - 0 0
 > > > 1 237 317
 > > > 1d81c5430 ---P--- 49 informix - 0 0
 > > > 1 197 329
 > > > 1d81c5c88 ---P--- 50 informix - 0 0
 > > > 1 165 552
 > > > 1d81c64e0 ---P--- 51 informix - 0 0
 > > > 2 202 252
 > > > 1d81c6d38 ---P--- 52 informix - 0 0
 > > > 1 304 76
 > > > 1d81c7590 ---P--- 53 informix - 0 0
 > > > 1 211 859
 > > > 1d81c7de8 ---P--- 54 informix - 0 0
 > > > 2 223 298
 > > > 1d81c8640 ---P--- 55 informix - 0 0
 > > > 1 204 414
 > > > 1d81c8e98 ---P--- 56 informix - 0 0
 > > > 1 263 329
 > > > 1d81c96f0 ---P--- 57 informix - 0 0
 > > > 2 237 364
 > > > 1d81c9f48 ---P--- 58 informix - 0 0
 > > > 2 302 174
 > > > 1d81ca7a0 ---P--- 136 informix - 0 0
 > > > 1 4 0
 > > > 1d81caff8 S--P--- 137 informix - 1d9fbc8f0 0
 > > > 1 186322 0
 > > > 1d81cb850 ---P--- 138 informix - 0 0
 > > > 0 272298 0
 > > > 1d81cc0a8 ---P--- 139 informix - 0 0
 > > > 1 44 0
 > > > 1d81cc900 S--P--- 142 informix - 1dbb72ee0 0
 > > > 0 0 211
 > > > 1d81cd158 Y--P--- 144 informix - 1db2aadc8 0
 > > > 1 9 23680
 > > > 1d81cd9b0 S--P--- 4692 informix - 1dbb72ee0 0
 > > > 2 3 4
 > > > 1d81ce208 S--P--- 4531 informix - 1dbb72ee0 0
 > > > 2 0 47
 > > > 1d81cea60 Y--P--- 235 saapmgr - 1def80778 0
 > > > 1 125 0
 > > > 1d81cf2b8 S--P--- 4693 informix - 1dbb72ee0 0
 > > > 2 3 1
 > > > 1d81d0bc0 S--P--- 151 informix - 1dbb72ee0 0
 > > > 2 13166 33392
 > > > 1d81d1418 S--P--- 152 informix - 1dbb72ee0 0
 > > > 2 14148 32374
 > > > 1d81d1c70 S--P--- 153 informix - 1dbb72ee0 0
 > > > 2 13705 31162
 > > > 1d81d2d20 ---P--- 155 informix - 0 0
 > > > 1 8890 0
 > > > 1d81d3dd0 Y--P--- 157 informix - 1db2aaca8 0
 > > > 2 13781 32324
 > > > 1d81d4628 Y--P--- 4827 informix - 1db2aaca8 0
 > > > 2 4 0
 > > > 1d81d4e80 S--P--- 159 informix - 1dbb72ee0 0
 > > > 2 13480 32593
 > > > 1d81d56d8 S--P--- 160 informix - 1dbb72ee0 0
 > > > 2 14200 32926
 > > > 1d81d5f30 Y--P--- 2693 saapmgr - 1e5a69148 0
 > > > 1 0 46
 > > > 1d81d6788 Y--P--- 173 informix - 1db2aadc8 0
 > > > 1 2 17280
 > > > 1d81d6fe0 Y--P--- 238 saapmgr - 1e029cca8 0
 > > > 1 0 0
 > > > 1d81d7838 Y--P--- 239 saapmgr - 1e3846028 0
 > > > 1 2 0
 > > > 1d81d8090 Y--P--- 237 saapmgr - 1dc95d4c8 0
 > > > 1 1 0
 > > > 1d81d88e8 Y--P--- 260 saapmgr - 1dcaccd78 0
 > > > 2 757 3042
 > > > 1d81d9140 Y--P--- 244 saapmgr - 1dfffeb78 0
 > > > 1 40 0
 > > > 1d81d9998 Y--P--- 258 saapmgr - 1e3176be0 0
 > > > 1 0 0
 > > > 1d81da1f0 Y--P--- 236 saapmgr - 1de8514c8 0
 > > > 1 1 0
 > > > 1d81daa48 Y--P--- 241 saapmgr - 1e0d85c28 0
 > > > 1 2 0
 > > > 1d81db2a0 Y--P--- 262 saapmgr - 1dcb5d538 0
 > > > 2 454 2717
 > > > 1d81dbaf8 Y--P--- 242 saapmgr - 1dcadf148 0
 > > > 1 0 0
 > > > 1d81dc350 Y--P--- 240 saapmgr - 1de777308 0
 > > > 1 1 0
 > > > 1d81dcba8 Y--P--- 254 saapmgr - 1dcb2f788 0
 > > > 1 3 0
 > > > 1d81dd400 Y--P--- 243 saapmgr - 1dd7a1df8 0
 > > > 1 1 0
 > > > 1d81ddc58 Y--P--- 259 saapmgr - 1dcb5d1d8 0
 > > > 2 820 2800
 > > > 1d81de4b0 Y--P--- 256 saapmgr - 1ddc9ac68 0
 > > > 1 9 0
 > > > 1d81ded08 Y--P--- 2647 wwwmgr - 1dd5c9708 0
 > > > 1 26 76
 > > > 1d81df560 Y--P--- 261 saapmgr - 1dd7ec2f8 0
 > > > 2 773 3510
 > > > 1d81dfdb8 Y--P--- 263 saapmgr - 1e12b86c0 0
 > > > 2 743 3245
 > > > 1d81e0610 Y--P--- 246 saapmgr - 1dd7c64a8 0
 > > > 1 0 0
 > > > 1d81e0e68 Y--P--- 264 saapmgr - 1e12b8ca8 0
 > > > 2 440 3917
 > > > 1d81e16c0 Y--P--- 265 saapmgr - 1dea7b3d0 0
 > > > 2 727 3044
 > > > 1d81e1f18 Y--P--- 266 saapmgr - 1dea7ba48 0
 > > > 1 460 3202
 > > > 1d81e2770 Y--P--- 267 saapmgr - 1e12b8f78 0
 > > > 2 651 3378
 > > > 1d81e2fc8 Y--P--- 268 saapmgr - 1e04de580 0
 > > > 1 891 3256
 > > > 1d81e3820 Y--P--- 269 saapmgr - 1e04debf8 0
 > > > 1 856 1164
 > > > 1d81e4078 Y--P--- 270 saapmgr - 1de6233d0 0
 > > > 1 2122 1273
 > > > 1d81e48d0 Y--P--- 271 saapmgr - 1e00966e8 0
 > > > 1 69 1045
 > > > 1d81e5128 Y--P--- 272 saapmgr - 1de623df0 0
 > > > 1 127 1460
 > > > 1d81e5980 Y--P--- 273 saapmgr - 1df259318 0
 > > > 1 63 1225
 > > > 1d81e61d8 Y--P--- 274 saapmgr - 1df259900 0
 > > > 1 65 1009
 > > > 1d81e6a30 Y--P--- 275 saapmgr - 1e028fa88 0
 > > > 1 86 1122
 > > > 1d81e7288 Y--P--- 276 saapmgr - 1e0324740 0
 > > > 1 73 1041
 > > > 1d81e7ae0 Y--P--- 277 saapmgr - 1e0324d28 0
 > > > 1 117 1273
 > > > 1d81e8338 Y--P--- 278 saapmgr - 1dcb43958 0
 > > > 1 69 1117
 > > > 1d81e8b90 Y--P--- 279 saapmgr - 1e317a630 0
 > > > 1 128223 247740
 > > > 1d81e93e8 Y--P--- 2737 saapmgr - 1dbb7f268 0
 > > > 1 0 26
 > > > 1d81e9c40 Y--P--- 4588 saapmgr - 1e182d428 0
 > > > 1 1 0
 > > > 1d81ea498 Y--P--- 282 saapmgr - 1dd7f96e8 0
 > > > 1 294 6235
 > > > 1d81eacf0 Y--P--- 283 saapmgr - 1dd7f9808 0
 > > > 1 186523 286151
 > > > 1d81eb548 Y--P--- 284 saapmgr - 1dd7b6788 0
 > > > 1 2224 337
 > > > 1d81ebda0 Y--P--- 285 saapmgr - 1dcb9bda8 0
 > > > 1 0 0
 > > > 1d81ec5f8 S--P--- 4473 informix - 1dbb72ee0 0
 > > > 6 4 140
 > > > 1d81ece50 S--P--- 4500 informix - 1dbb72ee0 0
 > > > 2 7 16
 > > > 1d81ed6a8 Y--P--- 288 saapmgr - 1e1638e38 0
 > > > 1 0 0
 > > > 1d81edf00 Y--P--- 289 saapmgr - 1de7e7028 0
 > > > 1 0 0
 > > > 1d81ee758 Y--P--- 290 saapmgr - 1e0cf49b8 0
 > > > 1 0 0
 > > > 1d81eefb0 Y--P--- 291 saapmgr - 1de713db0 0
 > > > 1 0 0
 > > > 1d81ef808 Y--P--- 292 saapmgr - 1df78b308 0
 > > > 1 0 23
 > > > 1d81f0060 Y--P--- 293 saapmgr - 1de9f9ef8 0
 > > > 1 0 51
 > > > 1d81f08b8 Y--P--- 297 saapmgr - 1e0886418 0
 > > > 1 5 0
 > > > 1d81f1110 Y--P--- 2738 saapmgr - 1dd875ce0 0
 > > > 1 0 0
 > > > 1d81f1968 Y--P--- 318 saapmgr - 1dfeff290 0
 > > > 1 94881 281121
 > > > 1d81f21c0 S--P--- 4393 informix - 1dbb72ee0 0
 > > > 2 15 438
 > > > 1d81f3270 Y--P--- 433 saapmgr - 1df83c278 0
 > > > 1 5 0
 > > > 1d81f3ac8 Y--P--- 1078 saapmgr - 1de046f18 0
 > > > 1 14 0
 > > > 1d81f4320 S--P--- 4828 informix - 1dbb72ee0 0
 > > > 2 0 5
 > > > 1d81f4b78 Y--P--- 4501 informix - 1db2aaca8 0
 > > > 2 7 18
 > > > 1e5116880 ---PR-- 5020 saapmgr - 0 0
 > > > 1 8901 0
 > > > 1e5117930 Y--P--- 2495 wwwmgr - 1dff80c28 0
 > > > 1 0 0
 > > > 1e5118188 L--P--- 4548 informix - 49a59840 120
 > > > 2 5 13
 > > > 1e51189e0 S--P--- 4376 informix - 1dbb72ee0 0
 > > > 2 97 451
 > > > 1e5119a90 Y--P--- 4430 informix - 1db2aaca8 0
 > > > 2 4 39
 > > > 1e511a2e8 S--P--- 4374 informix - 1dbb72ee0 0
 > > > 2 103 735
 > > > 1e511bbf0 Y--P--- 4427 informix - 1db2aaca8 0
 > > > 2 12 230
 > > > 1e511cca0 Y--P--- 5017 saapmgr - 1dd25f148 0
 > > > 1 7 43
 > > > 1e511d4f8 Y--P--- 2692 saapmgr - 1ee97e6a0 0
 > > > 1 0 18
 > > > 1e511dd50 Y--P--- 4448 informix - 1db2aaca8 0
 > > > 2 8 27
 > > > 1e511ee00 Y--P--- 4629 saapmgr - 1df4555f8 0
 > > > 1 0 0
 > > > 1e511f658 Y--P--- 4580 informix - 1db2aaca8 0
 > > > 2 4 13
 > > > 1e5120708 Y--P--- 3385 saapmgr - 1dbc6dad8 0
 > > > 1 0 0
 > > > 1e5122010 S--P--- 4506 informix - 1dbb72ee0 0
 > > > 2 10 279
 > > > 1e5122868 Y--P--- 5022 saapmgr - 1e0694bf8 0
 > > > 1 0 0
 > > > 1e5124170 Y--P--- 2739 saapmgr - 1ef31c028 0
 > > > 1 0 67
 > > > 1e51249c8 Y--P--- 4555 informix - 1db2aaca8 0
 > > > 2 5 16
 > > > 1e5125220 Y--P--- 2690 saapmgr - 1ee6c0d68 0
 > > > 1 0 51
 > > > 1e51262d0 Y--P--- 4372 informix - 1db2aaca8 0
 > > > 2 96 431
 > > > 1e5126b28 Y--P--- 2463 wwwmgr - 1ebf224a8 0
 > > > 1 6 0
 > > > 1e5127380 Y--P--- 2540 saapmgr - 1ed0f7cd8 0
 > > > 1 0 48
 > > > 1e5128430 S--P--- 4474 informix - 1dbb72ee0 0
 > > > 2 2 6
 > > > 1e5128c88 Y--P--- 2541 saapmgr - 1de9f16a0 0
 > > > 1 0 66
 > > > 1e5129d38 S--P--- 4394 informix - 1dbb72ee0 0
 > > > 2 25 32
 > > > 1e512b640 Y--P--- 4472 informix - 1db2aaca8 0
 > > > 5 8 15
 > > > 1e512be98 Y--P--- 4704 wwwmgr - 1e1a05d48 0
 > > > 1 46 17
 > > > 1e512cf48 Y--P--- 4565 informix - 1db2aaca8 0
 > > > 2 5 3
 > > > 1e512d7a0 S--P--- 4549 informix - 1dbb72ee0 0
 > > > 2 7 21
 > > > 1e512e850 S--P--- 4691 informix - 1dbb72ee0 0
 > > > 2 3 5
 > > > 1e512f0a8 Y--P--- 4829 informix - 1db2aaca8 0
 > > > 2 2 0
 > > > 1e512f900 S--P--- 4550 informix - 1dbb72ee0 0
 > > > 2 4 12
 > > > 1e5131208 S--P--- 4554 informix - 1dbb72ee0 0
 > > > 2 5 8
 > > > 1e5133368 Y--P--- 4371 informix - 1db2aaca8 0
 > > > 2 99 431
 > > > 1e5135d20 S--P--- 4637 informix - 1dbb72ee0 0
 > > > 2 7 5
 > > > 1e5136578 Y--P--- 4564 informix - 1db2aaca8 0
 > > > 2 7 11
 > > > 1e51386d8 Y--P--- 4628 saapmgr 1 1dbaedec8 0
 > > > 2 35202 0
 > > > 1e5139788 S--P--- 4553 informix - 1dbb72ee0 0
 > > > 2 4 140
 > > > 1e5139fe0 S--P--- 4446 informix - 1dbb72ee0 0
 > > > 2 6 191
 > > > 1e513c998 Y--P--- 4530 informix - 1db2aaca8 0
 > > > 2 7 13
 > > > 160 active, 256 total, 203 maximum concurrent
 > > >
 > > > --001a113ac4681904d00501107e85
 > > >
 > > >
 > > >
 > > >
 > >
 > >
 >
 *******************************************************************************
 > > > Forum Note: Use "Reply" to post a response in the discussion forum.
 > > >
 > > >
 > >
 > > --
 > > Fernando Nunes
 > > Portugal
 > >
 > > http://informix-technology.blogspot.com
 > > My email works... but I don't check it frequently...
 > >
 > > --20cf303bf57605a7b70501108c39
 > >
 > >
 > >
 > >
 >
 *******************************************************************************
 > > Forum Note: Use "Reply" to post a response in the discussion forum.
 > >
 > >
 >
 > --001a113a75a22a5ca7050110afc2
 >
 >
 >
 *******************************************************************************
 > Forum Note: Use "Reply" to post a response in the discussion forum.
 >
 
  
Messages In This Thread
 - wait for mutex
 FRANK -- Wednesday, 20 August 2014, at 10:33 a.m.
- Re: wait for mutex
 Fernando Nunes -- Wednesday, 20 August 2014, at 10:37 a.m.
- Re: wait for mutex
 FRANK -- Wednesday, 20 August 2014, at 10:47 a.m.
- Re: wait for mutex
 Madison Pruet -- Wednesday, 20 August 2014, at 11:05 a.m.
 - Re: wait for mutex
 JACQUES RENAUT -- Wednesday, 20 August 2014, at 11:41 a.m.
 - Re: wait for mutex
 Madison Pruet -- Wednesday, 20 August 2014, at 12:27 p.m.
  
  
 - RE: wait for mutex
 Paul Watson -- Wednesday, 20 August 2014, at 10:44 a.m.
  
  
IDS Forum is maintained by Administrator with WebBBS 5.12.
  | 
  |