While I was away our developers decided to do some cleanup of old records in our audit database. As I understand this there where deleting millions of old records when a lock halted their sql . once they cleared the other person who had the lock the logs haven't automatically rolled using the log_full.sh script.
They been keeping the system running while I was gone by doing a periodic ontape -a
Since this happened two days ago a full backup has happened twice (ontape -s -L 0) and the server was rebooted.
I'm not sure how to find the cause or fix this. Any ideas of what wrong and how to fix it?????
John David Adamski, Sr. Network Specialist Graceland University, 1 Univeristy Place, Lamoni, IA 50140 adamski@graceland.edu
Not sure if these logs might help
bar_dbug.log
2017-03-18 01:37:59 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object
table.
2017-03-18 01:38:00 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object
table.
2017-03-18 01:38:01 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object
table.
2017-03-18 01:38:01 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object
table.
2017-03-18 01:38:02 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object
table.
2017-03-18 01:38:02 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object
table.
2017-03-18 01:38:03 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object
table.
2017-03-18 04:10:58 2248 2245 smdbopen.c:117 nsmOpenDatabase: Count: opened = 8, missing = 0.
2017-03-18 04:10:58 2248 2245 smobject.c:1580 Error -88001: C-ISAM function 'dsmFetchDetail' retu
rned error (17).
2017-03-18 04:10:58 2248 2245 smtranid.c:176 Error -88001: C-ISAM function 'isread' returned erro
r (110).
2017-03-18 04:10:58 2248 2245 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object t
able.
2017-03-18 04:10:58 2248 2245 xbsa_so.c:1406 psmQueryObject: nsmOpenObjectCursor() returned 17
2017-03-18 04:10:58 2248 2245 smtranid.c:176 Error -88001: C-ISAM function 'isread' returned erro
r (110).
2017-03-18 04:12:10 3141 3139 smdbopen.c:117 nsmOpenDatabase: Count: opened = 8, missing = 0.
2017-03-18 04:12:11 3141 3139 smobject.c:1580 Error -88001: C-ISAM function 'dsmFetchDetail' retu
rned error (17).
2017-03-18 04:12:11 3141 3139 smtranid.c:176 Error -88001: C-ISAM function 'isread' returned erro
r (110).
2017-03-18 04:12:11 3141 3139 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object t
able.
informix cars: cat cars.log
Fri Mar 17 23:57:00 CDT 2017
Sat Mar 18 00:10:01 2017
00:10:01 Logical Log 402881 Complete, timestamp: 0x105918c7.
00:10:02 Logical Log 402861 - Backup Started
00:10:08 Logical Log 402861 - Backup Aborted
Aborted by client.
00:10:14 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402881 Complete, timestamp: 0x105918c7." "Logical Log 402881 Complete, timestamp: 0x105918c7." "" 23001
00:10:30 Logical Log 402882 Complete, timestamp: 0x105a30b1.
00:10:31 Logical Log 402861 - Backup Started
00:10:34 Logical Log 402861 - Backup Aborted
Aborted by client.
00:10:40 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402882 Complete, timestamp: 0x105a30b1." "Logical Log 402882 Complete, timestamp: 0x105a30b1." "" 23001
00:11:02 Logical Log 402883 Complete, timestamp: 0x105bb8af.
00:11:02 Logical Log 402861 - Backup Started
00:11:07 Logical Log 402861 - Backup Aborted
Aborted by client.
00:11:13 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402883 Complete, timestamp: 0x105bb8af." "Logical Log 402883 Complete, timestamp: 0x105bb8af." "" 23001
00:20:45 Logical Log 402884 Complete, timestamp: 0x10648e20.
00:20:46 Logical Log 402861 - Backup Started
00:20:50 Logical Log 402861 - Backup Aborted
Aborted by client.
00:20:56 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402884 Complete, timestamp: 0x10648e20." "Logical Log 402884 Complete, timestamp: 0x10648e20." "" 23001
00:34:21 Logical Log 402885 Complete, timestamp: 0x1069922a.
00:34:22 Logical Log 402861 - Backup Started
00:34:26 Logical Log 402861 - Backup Aborted
Aborted by client.
00:34:32 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402885 Complete, timestamp: 0x1069922a." "Logical Log 402885 Complete, timestamp: 0x1069922a." "" 23001
00:36:33 Logical Log 402886 Complete, timestamp: 0x106bca74.
00:36:33 Logical Log 402887 Complete, timestamp: 0x106bca74.
00:36:34 Logical Log 402861 - Backup Started
00:36:34 Logical Log 402888 Complete, timestamp: 0x106c6983.
00:36:37 Logical Log 402861 - Backup Aborted
Aborted by client.
00:36:41 Process exited with return code 152: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402887 Complete, timestamp: 0x106bca74." "Logical Log 402887 Complete, timestamp: 0x106bca74." "" 23001
00:36:42 Process exited with return code 152: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402888 Complete, timestamp: 0x106c6983." "Logical Log 402888 Complete, timestamp: 0x106c6983." "" 23001
00:36:43 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402886 Complete, timestamp: 0x106bca74." "Logical Log 402886 Complete, timestamp: 0x106bca74." "" 23001
00:49:13 Logical Log 402889 Complete, timestamp: 0x1071d576.
00:49:14 Logical Log 402861 - Backup Started
00:49:16 Logical Log 402861 - Backup Aborted
Aborted by client.
00:49:22 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402889 Complete, timestamp: 0x1071d576." "Logical Log 402889 Complete, timestamp: 0x1071d576." "" 23001
01:00:04 Logical Log 402890 Complete, timestamp: 0x107754cc.
01:00:06 Logical Log 402861 - Backup Started
01:00:11 Logical Log 402861 - Backup Aborted
Aborted by client.
01:00:15 Logical Log 402891 Complete, timestamp: 0x107869db.
01:00:15 Logical Log 402861 - Backup Started
01:00:16 Logical Log 402861 - Backup Aborted
Aborted by client.
01:00:17 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402890 Complete, timestamp: 0x107754cc." "Logical Log 402890 Complete, timestamp: 0x107754cc." "" 23001
01:00:20 Logical Log 402892 Complete, timestamp: 0x107939d5.
01:00:20 Logical Log 402861 - Backup Started
01:00:22 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402891 Complete, timestamp: 0x107869db." "Logical Log 402891 Complete, timestamp: 0x107869db." "" 23001
01:00:24 Logical Log 402893 Complete, timestamp: 0x1079f90e.
01:00:26 Logical Log 402861 - Backup Aborted
Aborted by client.
01:00:29 Logical Log 402894 Complete, timestamp: 0x107ac93e.
01:00:29 Logical Log 402861 - Backup Started
01:00:30 Process exited with return code 152: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402893 Complete, timestamp: 0x1079f90e." "Logical Log 402893 Complete, timestamp: 0x1079f90e." "" 23001
01:00:31 Logical Log 402861 - Backup Aborted
Aborted by client.
01:00:32 Logical Log 402895 Complete, timestamp: 0x107b6145.
01:00:32 Logical Log 402861 - Backup Started
01:00:32 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402892 Complete, timestamp: 0x107939d5." "Logical Log 402892 Complete, timestamp: 0x107939d5." "" 23001
01:00:35 Logical Log 402861 - Backup Aborted
Aborted by client.
01:00:36 Logical Log 402896 Complete, timestamp: 0x107c4ee2.
01:00:37 Logical Log 402861 - Backup Started
01:00:37 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402894 Complete, timestamp: 0x107ac93e." "Logical Log 402894 Complete, timestamp: 0x107ac93e." "" 23001
01:00:39 Logical Log 402861 - Backup Aborted
Aborted by client.
01:00:41 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402895 Complete, timestamp: 0x107b6145." "Logical Log 402895 Complete, timestamp: 0x107b6145." "" 23001
01:00:45 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402896 Complete, timestamp: 0x107c4ee2." "Logical Log 402896 Complete, timestamp: 0x107c4ee2." "" 23001
01:00:47 Logical Log 402897 Complete, timestamp: 0x107d387c.
01:00:47 Logical Log 402861 - Backup Started
01:00:49 Logical Log 402861 - Backup Aborted
Aborted by client.
01:00:52 Logical Log 402898 Complete, timestamp: 0x107e0cb9.
01:00:53 Logical Log 402861 - Backup Started
01:00:54 Logical Log 402861 - Backup Aborted
Aborted by client.
01:00:55 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402897 Complete, timestamp: 0x107d387c." "Logical Log 402897 Complete, timestamp: 0x107d387c." "" 23001
01:00:56 Logical Log 402899 Complete, timestamp: 0x107eb391.
01:00:56 Logical Log 402861 - Backup Started
01:00:58 Logical Log 402861 - Backup Aborted
Aborted by client.
01:01:00 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402898 Complete, timestamp: 0x107e0cb9." "Logical Log 402898 Complete, timestamp: 0x107e0cb9." "" 23001
01:01:00 Logical Log 402900 Complete, timestamp: 0x107f7b85.
01:01:00 Logical Log 402861 - Backup Started
01:01:01 Logical Log 402861 - Backup Aborted
Aborted by client.
01:01:04 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402899 Complete, timestamp: 0x107eb391." "Logical Log 402899 Complete, timestamp: 0x107eb391." "" 23001
01:01:05 Logical Log 402901 Complete, timestamp: 0x10804e3c.
01:01:05 Logical Log 402861 - Backup Started
01:01:07 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402900 Complete, timestamp: 0x107f7b85." "Logical Log 402900 Complete, timestamp: 0x107f7b85." "" 23001
01:01:14 Logical Log 402861 - Backup Aborted
Aborted by client.
01:01:20 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402901 Complete, timestamp: 0x10804e3c." "Logical Log 402901 Complete, timestamp: 0x10804e3c." "" 23001
01:08:46 Logical Log 402902 Complete, timestamp: 0x10846f59.
01:08:49 Logical Log 402861 - Backup Started
01:08:50 Logical Log 402861 - Backup Aborted
Aborted by client.
01:08:56 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402902 Complete, timestamp: 0x10846f59." "Logical Log 402902 Complete, timestamp: 0x10846f59." "" 23001
01:11:25 Logical Log 402903 Complete, timestamp: 0x10851e4f.
01:11:25 Logical Log 402861 - Backup Started
01:11:33 Logical Log 402861 - Backup Aborted
Aborted by client.
01:11:39 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402903 Complete, timestamp: 0x10851e4f." "Logical Log 402903 Complete, timestamp: 0x10851e4f." "" 23001
01:11:42 Logical Log 402904 Complete, timestamp: 0x1085c8db.
01:11:43 Logical Log 402861 - Backup Started
01:11:44 Logical Log 402861 - Backup Aborted
Aborted by client.
01:11:50 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402904 Complete, timestamp: 0x1085c8db." "Logical Log 402904 Complete, timestamp: 0x1085c8db." "" 23001
01:13:41 Logical Log 402905 Complete, timestamp: 0x10868e3e.
01:13:41 Logical Log 402861 - Backup Started
01:13:47 Logical Log 402861 - Backup Aborted
Aborted by client.
01:13:53 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402905 Complete, timestamp: 0x10868e3e." "Logical Log 402905 Complete, timestamp: 0x10868e3e." "" 23001
01:14:30 Logical Log 402906 Complete, timestamp: 0x10872c35.
01:14:30 Logical Log 402861 - Backup Started
01:14:32 Logical Log 402861 - Backup Aborted
Aborted by client.
01:14:38 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402906 Complete, timestamp: 0x10872c35." "Logical Log 402906 Complete, timestamp: 0x10872c35." "" 23001
01:15:19 Logical Log 402907 Complete, timestamp: 0x1087c7ea.
01:15:19 Logical Log 402861 - Backup Started
01:15:20 Logical Log 402861 - Backup Aborted
Aborted by client.
01:15:26 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402907 Complete, timestamp: 0x1087c7ea." "Logical Log 402907 Complete, timestamp: 0x1087c7ea." "" 23001
01:15:31 Logical Log 402908 Complete, timestamp: 0x10885912.
01:15:31 Logical Log 402861 - Backup Started
01:15:33 Logical Log 402861 - Backup Aborted
Aborted by client.
01:15:39 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402908 Complete, timestamp: 0x10885912." "Logical Log 402908 Complete, timestamp: 0x10885912." "" 23001
01:16:24 Logical Log 402909 Complete, timestamp: 0x108929d4.
01:16:25 Logical Log 402861 - Backup Started
01:16:26 Logical Log 402861 - Backup Aborted
Aborted by client.
01:16:27 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402909 Complete, timestamp: 0x108929d4." "Logical Log 402909 Complete, timestamp: 0x108929d4." "" 23001
01:17:12 Logical Log Files are Full -- Backup is Needed
04:07:00 Logical Log 402861 - Backup Started
04:07:03 Logical Log 402861 - Backup Completed
04:07:03 Logical Log 402862 - Backup Started
04:07:04 Logical Log 402862 - Backup Completed
04:07:04 Logical Log 402863 - Backup Started
04:07:04 Logical Log 402910 Complete, timestamp: 0x108ab32c.
04:07:05 Logical Log 402863 - Backup Completed
04:07:05 Logical Log 402864 - Backup Started
04:07:05 Logical Log 402864 - Backup Completed
04:07:05 Logical Log 402865 - Backup Started
04:07:05 Logical Log 402865 - Backup Completed
04:07:06 Logical Log 402866 - Backup Started
04:07:06 Logical Log 402866 - Backup Completed
04:07:06 Logical Log 402867 - Backup Started
04:07:07 Logical Log 402867 - Backup Completed
04:07:07 Logical Log 402868 - Backup Started
04:07:08 Logical Log 402868 - Backup Completed
04:07:08 Logical Log 402869 - Backup Started
04:07:08 Logical Log 402869 - Backup Completed
04:07:08 Logical Log 402870 - Backup Started
04:07:09 Logical Log 402870 - Backup Completed
04:07:09 Logical Log 402871 - Backup Started
04:07:09 Logical Log 402871 - Backup Completed
04:07:09 Logical Log 402872 - Backup Started
04:07:10 Logical Log 402872 - Backup Completed
04:07:10 Logical Log 402873 - Backup Started
04:07:11 Logical Log 402873 - Backup Completed
04:07:11 Process exited with return code 152: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402910 Complete, timestamp: 0x108ab32c." "Logical Log 402910 Complete, timestamp: 0x108ab32c." "" 23001
04:07:11 Logical Log 402874 - Backup Started
04:07:11 Logical Log 402874 - Backup Completed
04:07:11 Logical Log 402875 - Backup Started
04:07:11 Logical Log 402875 - Backup Completed
04:07:11 Logical Log 402876 - Backup Started
04:07:12 Logical Log 402876 - Backup Completed
04:07:12 Logical Log 402877 - Backup Started
04:07:12 Logical Log 402877 - Backup Completed
04:07:12 Logical Log 402878 - Backup Started
04:07:12 Logical Log 402878 - Backup Completed
04:07:12 Logical Log 402879 - Backup Started
04:07:13 Logical Log 402879 - Backup Completed
04:07:13 Logical Log 402880 - Backup Started
04:07:13 Logical Log 402880 - Backup Completed
04:07:13 Logical Log 402881 - Backup Started
04:07:13 Logical Log 402881 - Backup Completed
04:07:13 Logical Log 402882 - Backup Started
04:07:14 Logical Log 402882 - Backup Completed
04:07:14 Logical Log 402883 - Backup Started
04:07:14 Logical Log 402883 - Backup Completed
04:07:14 Logical Log 402884 - Backup Started
04:07:14 Logical Log 402884 - Backup Completed
04:07:14 Logical Log 402885 - Backup Started
04:07:14 Logical Log 402885 - Backup Completed
04:07:14 Logical Log 402886 - Backup Started
04:07:15 Logical Log 402886 - Backup Completed
04:07:15 Logical Log 402887 - Backup Started
04:07:15 Logical Log 402887 - Backup Completed
04:07:15 Logical Log 402888 - Backup Started
04:07:15 Logical Log 402888 - Backup Completed
04:07:15 Logical Log 402889 - Backup Started
04:07:15 Logical Log 402889 - Backup Completed
04:07:16 Logical Log 402890 - Backup Started
04:07:16 Logical Log 402890 - Backup Completed
04:07:16 Logical Log 402891 - Backup Started
04:07:16 Logical Log 402891 - Backup Completed
04:07:16 Logical Log 402892 - Backup Started
04:07:17 Logical Log 402892 - Backup Completed
04:07:17 Logical Log 402893 - Backup Started
04:07:17 Logical Log 402893 - Backup Completed
04:07:17 Logical Log 402894 - Backup Started
04:07:17 Logical Log 402894 - Backup Completed
04:07:17 Logical Log 402895 - Backup Started
04:07:18 Logical Log 402895 - Backup Completed
04:07:18 Logical Log 402896 - Backup Started
04:07:18 Logical Log 402896 - Backup Completed
04:07:18 Logical Log 402897 - Backup Started
04:07:18 Logical Log 402897 - Backup Completed
04:07:19 Logical Log 402898 - Backup Started
04:07:19 Logical Log 402898 - Backup Completed
04:07:19 Logical Log 402899 - Backup Started
04:07:19 Logical Log 402899 - Backup Completed
04:07:19 Logical Log 402900 - Backup Started
04:07:19 Logical Log 402900 - Backup Completed
04:07:19 Logical Log 402901 - Backup Started
04:07:20 Logical Log 402901 - Backup Completed
04:07:20 Logical Log 402902 - Backup Started
04:07:20 Logical Log 402902 - Backup Completed
04:07:20 Logical Log 402903 - Backup Started
04:07:20 Logical Log 402903 - Backup Completed
04:07:20 Logical Log 402904 - Backup Started
04:07:21 Logical Log 402904 - Backup Completed
04:07:21 Logical Log 402905 - Backup Started
04:07:21 Logical Log 402905 - Backup Completed
04:07:21 Logical Log 402906 - Backup Started
04:07:21 Logical Log 402906 - Backup Completed
04:07:21 Logical Log 402907 - Backup Started
04:07:22 Logical Log 402907 - Backup Completed
04:07:22 Logical Log 402908 - Backup Started
04:07:22 Logical Log 402908 - Backup Completed
04:07:22 Logical Log 402909 - Backup Started
04:07:22 Logical Log 402909 - Backup Completed
04:07:23 Logical Log 402910 - Backup Started
04:07:23 Logical Log 402910 - Backup Completed
04:07:38 Logical Log 402911 - Backup Started
04:07:39 Logical Log 402911 - Backup Completed
04:07:39 Logical Log 402911 Complete, timestamp: 0x108d3f46.
04:07:40 Logical Log 402912 Complete, timestamp: 0x108e620a.
04:07:47 Logical Log 402913 Complete, timestamp: 0x108fcaec.
04:07:47 Process exited with return code 152: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402911 Complete, timestamp: 0x108d3f46." "Logical Log 402911 Complete, timestamp: 0x108d3f46." "" 23001
04:07:47 Process exited with return code 152: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402912 Complete, timestamp: 0x108e620a." "Logical Log 402912 Complete, timestamp: 0x108e620a." "" 23001
04:07:54 Process exited with return code 152: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402913 Complete, timestamp: 0x108fcaec." "Logical Log 402913 Complete, timestamp: 0x108fcaec." "" 23001
04:10:57 Logical Log 402914 Complete, timestamp: 0x109646d9.
04:10:58 Logical Log 402912 - Backup Started
04:10:59 Logical Log 402912 - Backup Aborted
Aborted by client.
04:11:05 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402914 Complete, timestamp: 0x109646d9." "Logical Log 402914 Complete, timestamp: 0x109646d9." "" 23001
04:12:10 Logical Log 402915 Complete, timestamp: 0x10a02bbd.
04:12:10 Logical Log 402912 - Backup Started
04:12:11 Logical Log 402912 - Backup Aborted
Aborted by client.
04:12:15 Logical Log 402916 Complete, timestamp: 0x10a08ade.
04:12:16 Logical Log 402912 - Backup Started
04:12:17 Logical Log 402912 - Backup Aborted
Aborted by client.