== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 03:12:39 (1782630759) start mds service on oleg231-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg231-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg231-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 433 49567 1% /mnt/lustre[OST:0] filesystem_summary: 50096 529 49567 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1545 (time 1782630806.01 total 10.01 last 154.41) - open/close/unlink 3214 (time 1782630816.01 total 20.01 last 166.89) - open/close/unlink 4906 (time 1782630826.01 total 30.01 last 169.12) - open/close/unlink 6330 (time 1782630836.02 total 40.02 last 142.31) - open/close/unlink 7914 (time 1782630846.02 total 50.02 last 158.39) - open/close/unlink 9881 (time 1782630856.02 total 60.02 last 196.66) - open/close/unlink 11760 (time 1782630866.02 total 70.02 last 187.89) - open/close/unlink 13916 (time 1782630876.03 total 80.03 last 215.50) - open/close/unlink 16069 (time 1782630886.03 total 90.03 last 215.25) - open/close/unlink 18138 (time 1782630896.04 total 100.04 last 206.64) - open/close/unlink 20000 (time 1782630904.71 total 108.70 last 214.93) - open/close/unlink 22129 (time 1782630914.71 total 118.71 last 212.83) - open/close/unlink 24354 (time 1782630924.71 total 128.71 last 222.49) - open/close/unlink 26521 (time 1782630934.71 total 138.71 last 216.69) - open/close/unlink 28637 (time 1782630944.71 total 148.71 last 211.56) - open/close/unlink 30000 (time 1782630951.34 total 155.33 last 205.72) - open/close/unlink 31924 (time 1782630961.34 total 165.34 last 192.33) - open/close/unlink 33937 (time 1782630971.34 total 175.34 last 201.28) - open/close/unlink 36031 (time 1782630981.35 total 185.34 last 209.33) - open/close/unlink 38003 (time 1782630991.35 total 195.34 last 197.20) - open/close/unlink 40000 (time 1782631000.42 total 204.42 last 220.13) - open/close/unlink 42125 (time 1782631010.42 total 214.42 last 212.45) - open/close/unlink 44152 (time 1782631020.42 total 224.42 last 202.68) - open/close/unlink 46094 (time 1782631030.42 total 234.42 last 194.19) - open/close/unlink 48246 (time 1782631040.44 total 244.44 last 214.84) - open/close/unlink 50000 (time 1782631049.49 total 253.48 last 193.90) - open/close/unlink 52011 (time 1782631059.49 total 263.49 last 201.02) - open/close/unlink 54063 (time 1782631069.49 total 273.49 last 205.17) - open/close/unlink 55993 (time 1782631079.49 total 283.49 last 192.93) - open/close/unlink 58120 (time 1782631089.50 total 293.50 last 212.59) - open/close/unlink 60000 (time 1782631099.23 total 303.22 last 193.26) - open/close/unlink 62010 (time 1782631109.24 total 313.23 last 200.80) - open/close/unlink 64114 (time 1782631119.24 total 323.23 last 210.38) total: 64768 open/close/unlink in 326.01 seconds: 198.67 ops/second - open/close/unlink 1556 (time 1782630806.05 total 10.00 last 155.60) - open/close/unlink 3223 (time 1782630816.06 total 20.00 last 166.67) - open/close/unlink 4927 (time 1782630826.06 total 30.00 last 170.35) - open/close/unlink 6351 (time 1782630836.06 total 40.01 last 142.33) - open/close/unlink 7925 (time 1782630846.07 total 50.01 last 157.36) - open/close/unlink 9874 (time 1782630856.07 total 60.02 last 194.84) - open/close/unlink 11765 (time 1782630866.07 total 70.02 last 189.05) - open/close/unlink 13946 (time 1782630876.08 total 80.02 last 218.03) - open/close/unlink 16115 (time 1782630886.08 total 90.03 last 216.81) - open/close/unlink 18180 (time 1782630896.08 total 100.03 last 206.42) - open/close/unlink 20000 (time 1782630904.45 total 108.39 last 217.63) - open/close/unlink 22132 (time 1782630914.45 total 118.39 last 213.20) - open/close/unlink 24345 (time 1782630924.45 total 128.39 last 221.26) - open/close/unlink 26518 (time 1782630934.45 total 138.40 last 217.24) - open/close/unlink 28637 (time 1782630944.45 total 148.40 last 211.88) - open/close/unlink 30000 (time 1782630950.99 total 154.93 last 208.64) - open/close/unlink 31893 (time 1782630960.99 total 164.94 last 189.20) - open/close/unlink 33891 (time 1782630970.99 total 174.94 last 199.76) - open/close/unlink 35945 (time 1782630981.00 total 184.95 last 205.22) - open/close/unlink 37907 (time 1782630991.00 total 194.95 last 196.15) - open/close/unlink 40000 (time 1782631000.55 total 204.49 last 219.36) - open/close/unlink 42130 (time 1782631010.55 total 214.49 last 212.94) - open/close/unlink 44149 (time 1782631020.55 total 224.50 last 201.83) - open/close/unlink 46084 (time 1782631030.55 total 234.50 last 193.48) - open/close/unlink 48256 (time 1782631040.56 total 244.50 last 217.13) - open/close/unlink 50000 (time 1782631049.59 total 253.53 last 193.12) - open/close/unlink 52008 (time 1782631059.59 total 263.54 last 200.73) - open/close/unlink 54048 (time 1782631069.59 total 273.54 last 203.98) - open/close/unlink 55971 (time 1782631079.59 total 283.54 last 192.26) - open/close/unlink 58091 (time 1782631089.60 total 293.54 last 211.92) - open/close/unlink 60000 (time 1782631099.42 total 303.36 last 194.41) - open/close/unlink 61989 (time 1782631109.42 total 313.37 last 198.81) - open/close/unlink 64083 (time 1782631119.42 total 323.37 last 209.33) total: 64768 open/close/unlink in 327.75 seconds: 197.62 ops/second - open/close/unlink 1519 (time 1782630805.72 total 10.01 last 151.81) - open/close/unlink 3179 (time 1782630815.73 total 20.01 last 165.92) - open/close/unlink 4881 (time 1782630825.73 total 30.01 last 170.16) - open/close/unlink 6302 (time 1782630835.73 total 40.01 last 142.08) - open/close/unlink 7856 (time 1782630845.73 total 50.02 last 155.35) - open/close/unlink 9834 (time 1782630855.73 total 60.02 last 197.79) - open/close/unlink 11707 (time 1782630865.73 total 70.02 last 187.30) - open/close/unlink 13867 (time 1782630875.74 total 80.02 last 215.91) - open/close/unlink 15981 (time 1782630885.74 total 90.03 last 211.32) - open/close/unlink 18077 (time 1782630895.74 total 100.03 last 209.56) - open/close/unlink 20000 (time 1782630904.74 total 109.03 last 213.66) - open/close/unlink 22141 (time 1782630914.75 total 119.03 last 214.06) - open/close/unlink 24395 (time 1782630924.75 total 129.03 last 225.40) - open/close/unlink 26543 (time 1782630934.75 total 139.03 last 214.76) - open/close/unlink 28656 (time 1782630944.75 total 149.03 last 211.26) - open/close/unlink 30000 (time 1782630951.17 total 155.46 last 209.22) - open/close/unlink 31906 (time 1782630961.18 total 165.46 last 190.59) - open/close/unlink 33895 (time 1782630971.18 total 175.46 last 198.82) - open/close/unlink 35951 (time 1782630981.18 total 185.46 last 205.58) - open/close/unlink 37913 (time 1782630991.18 total 195.47 last 196.16) - open/close/unlink 40000 (time 1782631000.74 total 205.02 last 218.46) - open/close/unlink 42130 (time 1782631010.74 total 215.02 last 212.98) - open/close/unlink 44154 (time 1782631020.74 total 225.02 last 202.35) - open/close/unlink 46094 (time 1782631030.74 total 235.02 last 193.99) - open/close/unlink 48260 (time 1782631040.74 total 245.03 last 216.54) - open/close/unlink 50000 (time 1782631049.86 total 254.14 last 190.83) - open/close/unlink 52017 (time 1782631059.86 total 264.14 last 201.67) - open/close/unlink 54046 (time 1782631069.86 total 274.15 last 202.86) - open/close/unlink 55976 (time 1782631079.86 total 284.15 last 193.00) - open/close/unlink 58115 (time 1782631089.87 total 294.15 last 213.84) - open/close/unlink 60000 (time 1782631099.66 total 303.95 last 192.43) - open/close/unlink 61972 (time 1782631109.66 total 313.95 last 197.18) - open/close/unlink 64088 (time 1782631119.67 total 323.95 last 211.54) total: 64768 open/close/unlink in 329.13 seconds: 196.79 ops/second - open/close/unlink 1548 (time 1782630806.05 total 10.00 last 154.79) - open/close/unlink 3213 (time 1782630816.05 total 20.00 last 166.45) - open/close/unlink 4910 (time 1782630826.06 total 30.01 last 169.57) - open/close/unlink 6334 (time 1782630836.06 total 40.01 last 142.38) - open/close/unlink 7923 (time 1782630846.07 total 50.02 last 158.84) - open/close/unlink 9876 (time 1782630856.07 total 60.02 last 195.18) - open/close/unlink 11760 (time 1782630866.08 total 70.03 last 188.31) - open/close/unlink 13936 (time 1782630876.08 total 80.03 last 217.52) - open/close/unlink 16106 (time 1782630886.08 total 90.03 last 216.95) - open/close/unlink 18177 (time 1782630896.09 total 100.04 last 207.01) - open/close/unlink 20000 (time 1782630904.55 total 108.50 last 215.30) - open/close/unlink 22130 (time 1782630914.56 total 118.51 last 212.92) - open/close/unlink 24371 (time 1782630924.56 total 128.51 last 224.09) - open/close/unlink 26520 (time 1782630934.56 total 138.51 last 214.84) - open/close/unlink 28624 (time 1782630944.56 total 148.51 last 210.34) - open/close/unlink 30000 (time 1782630951.34 total 155.29 last 203.01) - open/close/unlink 31878 (time 1782630961.34 total 165.29 last 187.77) - open/close/unlink 33869 (time 1782630971.35 total 175.30 last 199.03) - open/close/unlink 35941 (time 1782630981.35 total 185.30 last 207.14) - open/close/unlink 37911 (time 1782630991.35 total 195.30 last 196.93) - open/close/unlink 40000 (time 1782631000.89 total 204.84 last 219.11) - open/close/unlink 42111 (time 1782631010.89 total 214.84 last 211.05) - open/close/unlink 44151 (time 1782631020.89 total 224.84 last 203.97) - open/close/unlink 46079 (time 1782631030.89 total 234.84 last 192.79) - open/close/unlink 48258 (time 1782631040.89 total 244.84 last 217.89) - open/close/unlink 50000 (time 1782631050.10 total 254.05 last 189.19) - open/close/unlink 52032 (time 1782631060.10 total 264.06 last 203.11) - open/close/unlink 54052 (time 1782631070.11 total 274.06 last 201.96) - open/close/unlink 56000 (time 1782631080.11 total 284.06 last 194.71) - open/close/unlink 58127 (time 1782631090.11 total 294.06 last 212.67) - open/close/unlink 60000 (time 1782631099.76 total 303.71 last 194.10) - open/close/unlink 62004 (time 1782631109.76 total 313.71 last 200.37) - open/close/unlink 64111 (time 1782631119.77 total 323.72 last 210.62) total: 64768 open/close/unlink in 328.98 seconds: 196.87 ops/second - open/close/unlink 1543 (time 1782630806.07 total 10.00 last 154.25) - open/close/unlink 3199 (time 1782630816.07 total 20.01 last 165.55) - open/close/unlink 4886 (time 1782630826.07 total 30.01 last 168.66) - open/close/unlink 6315 (time 1782630836.07 total 40.01 last 142.88) - open/close/unlink 7906 (time 1782630846.08 total 50.01 last 159.07) - open/close/unlink 9864 (time 1782630856.08 total 60.02 last 195.73) - open/close/unlink 11736 (time 1782630866.08 total 70.02 last 187.20) - open/close/unlink 13885 (time 1782630876.08 total 80.02 last 214.88) - open/close/unlink 16024 (time 1782630886.09 total 90.02 last 213.77) - open/close/unlink 18090 (time 1782630896.09 total 100.03 last 206.54) - open/close/unlink 20000 (time 1782630905.00 total 108.94 last 214.34) - open/close/unlink 22110 (time 1782630915.00 total 118.94 last 210.93) - open/close/unlink 24323 (time 1782630925.01 total 128.95 last 221.18) - open/close/unlink 26470 (time 1782630935.01 total 138.95 last 214.61) - open/close/unlink 28577 (time 1782630945.02 total 148.95 last 210.62) - open/close/unlink 30000 (time 1782630951.99 total 155.93 last 204.08) - open/close/unlink 31868 (time 1782630961.99 total 165.93 last 186.76) - open/close/unlink 33894 (time 1782630971.99 total 175.93 last 202.60) - open/close/unlink 35963 (time 1782630981.99 total 185.93 last 206.87) - open/close/unlink 37928 (time 1782630992.00 total 195.93 last 196.45) - open/close/unlink 40000 (time 1782631001.46 total 205.39 last 219.01) - open/close/unlink 42084 (time 1782631011.46 total 215.40 last 208.32) - open/close/unlink 44122 (time 1782631021.46 total 225.40 last 203.75) - open/close/unlink 46055 (time 1782631031.46 total 235.40 last 193.29) - open/close/unlink 48230 (time 1782631041.47 total 245.40 last 217.47) - open/close/unlink 50000 (time 1782631050.77 total 254.70 last 190.28) - open/close/unlink 52046 (time 1782631060.77 total 264.71 last 204.57) - open/close/unlink 54063 (time 1782631070.77 total 274.71 last 201.62) - open/close/unlink 56033 (time 1782631080.78 total 284.71 last 196.93) - open/close/unlink 58179 (time 1782631090.78 total 294.71 last 214.56) - open/close/unlink 60000 (time 1782631100.41 total 304.34 last 189.12) - open/close/unlink 62005 (time 1782631110.41 total 314.35 last 200.43) - open/close/unlink 64131 (time 1782631120.41 total 324.35 last 212.58) total: 64768 open/close/unlink in 330.19 seconds: 196.15 ops/second Waiting for MDT destroys to complete oleg231-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg231-server unloading modules via unload_modules_local on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local oleg231-server: modules unloaded. pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2