== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 00:09:17 (1786939757) start mds service on oleg338-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg338-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg338-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0000 oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 433 49567 1% /mnt/lustre[OST:0] filesystem_summary: 50096 529 49567 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 868 (time 1786939808.96 total 10.01 last 86.74) - open/close/unlink 1972 (time 1786939818.99 total 20.04 last 110.03) - open/close/unlink 3425 (time 1786939829.02 total 30.07 last 144.83) - open/close/unlink 4829 (time 1786939839.02 total 40.07 last 140.38) - open/close/unlink 5997 (time 1786939849.03 total 50.08 last 116.77) - open/close/unlink 7270 (time 1786939859.03 total 60.08 last 127.27) - open/close/unlink 9104 (time 1786939869.03 total 70.08 last 183.32) - open/close/unlink 10000 (time 1786939873.53 total 74.58 last 199.40) - open/close/unlink 11826 (time 1786939883.53 total 84.58 last 182.54) - open/close/unlink 13532 (time 1786939893.53 total 94.58 last 170.55) - open/close/unlink 15055 (time 1786939903.53 total 104.58 last 152.30) - open/close/unlink 16832 (time 1786939913.53 total 114.58 last 177.69) - open/close/unlink 18506 (time 1786939923.54 total 124.59 last 167.38) - open/close/unlink 20000 (time 1786939931.57 total 132.62 last 185.93) - open/close/unlink 21825 (time 1786939941.57 total 142.62 last 182.48) - open/close/unlink 23806 (time 1786939951.57 total 152.62 last 198.06) - open/close/unlink 25841 (time 1786939961.58 total 162.63 last 203.36) - open/close/unlink 27721 (time 1786939971.58 total 172.63 last 187.97) - open/close/unlink 29857 (time 1786939981.58 total 182.63 last 213.60) - open/close/unlink 32053 (time 1786939991.58 total 192.63 last 219.60) - open/close/unlink 34279 (time 1786940001.58 total 202.63 last 222.59) - open/close/unlink 36611 (time 1786940011.59 total 212.64 last 233.10) - open/close/unlink 38943 (time 1786940021.59 total 222.64 last 233.13) - open/close/unlink 40000 (time 1786940025.75 total 226.80 last 253.92) - open/close/unlink 42352 (time 1786940035.75 total 236.80 last 235.18) - open/close/unlink 44547 (time 1786940045.76 total 246.80 last 219.48) - open/close/unlink 46936 (time 1786940055.76 total 256.81 last 238.86) - open/close/unlink 49304 (time 1786940065.76 total 266.81 last 236.75) - open/close/unlink 50000 (time 1786940069.06 total 270.11 last 211.03) - open/close/unlink 52328 (time 1786940079.06 total 280.11 last 232.77) - open/close/unlink 54731 (time 1786940089.06 total 290.11 last 240.25) - open/close/unlink 56929 (time 1786940099.07 total 300.12 last 219.67) - open/close/unlink 59085 (time 1786940109.07 total 310.12 last 215.58) - open/close/unlink 60000 (time 1786940112.87 total 313.91 last 240.92) - open/close/unlink 62268 (time 1786940122.87 total 323.92 last 226.77) - open/close/unlink 64597 (time 1786940132.87 total 333.92 last 232.85) total: 64768 open/close/unlink in 334.71 seconds: 193.50 ops/second - open/close/unlink 875 (time 1786939809.05 total 10.00 last 87.46) - open/close/unlink 1974 (time 1786939819.06 total 20.01 last 109.84) - open/close/unlink 3410 (time 1786939829.06 total 30.01 last 143.59) - open/close/unlink 4810 (time 1786939839.06 total 40.01 last 139.95) - open/close/unlink 5975 (time 1786939849.07 total 50.02 last 116.40) - open/close/unlink 7258 (time 1786939859.07 total 60.03 last 128.27) - open/close/unlink 9075 (time 1786939869.10 total 70.06 last 181.11) - open/close/unlink 10000 (time 1786939873.75 total 74.70 last 199.20) - open/close/unlink 11816 (time 1786939883.75 total 84.71 last 181.53) - open/close/unlink 13542 (time 1786939893.76 total 94.71 last 172.49) - open/close/unlink 15092 (time 1786939903.76 total 104.72 last 154.95) - open/close/unlink 16884 (time 1786939913.76 total 114.72 last 179.20) - open/close/unlink 18530 (time 1786939923.76 total 124.72 last 164.53) - open/close/unlink 20000 (time 1786939931.65 total 132.61 last 186.30) - open/close/unlink 21847 (time 1786939941.66 total 142.61 last 184.68) - open/close/unlink 23801 (time 1786939951.66 total 152.61 last 195.35) - open/close/unlink 25791 (time 1786939961.66 total 162.61 last 199.00) - open/close/unlink 27655 (time 1786939971.66 total 172.62 last 186.34) - open/close/unlink 29784 (time 1786939981.66 total 182.62 last 212.85) - open/close/unlink 31982 (time 1786939991.67 total 192.62 last 219.72) - open/close/unlink 34184 (time 1786940001.67 total 202.62 last 220.16) - open/close/unlink 36496 (time 1786940011.67 total 212.63 last 231.11) - open/close/unlink 38905 (time 1786940021.67 total 222.63 last 240.90) - open/close/unlink 40000 (time 1786940026.17 total 227.12 last 243.60) - open/close/unlink 42396 (time 1786940036.17 total 237.13 last 239.56) - open/close/unlink 44624 (time 1786940046.17 total 247.13 last 222.77) - open/close/unlink 47024 (time 1786940056.17 total 257.13 last 239.95) - open/close/unlink 49342 (time 1786940066.18 total 267.13 last 231.77) - open/close/unlink 50000 (time 1786940069.31 total 270.26 last 210.07) - open/close/unlink 52328 (time 1786940079.31 total 280.26 last 232.76) - open/close/unlink 54710 (time 1786940089.31 total 290.27 last 238.09) - open/close/unlink 56932 (time 1786940099.32 total 300.27 last 222.09) - open/close/unlink 59097 (time 1786940109.32 total 310.28 last 216.47) - open/close/unlink 60000 (time 1786940113.04 total 313.99 last 242.96) - open/close/unlink 62289 (time 1786940123.04 total 323.99 last 228.86) - open/close/unlink 64605 (time 1786940133.04 total 334.00 last 231.53) total: 64768 open/close/unlink in 334.98 seconds: 193.35 ops/second - open/close/unlink 877 (time 1786939808.89 total 10.01 last 87.64) - open/close/unlink 1952 (time 1786939818.90 total 20.01 last 107.46) - open/close/unlink 3399 (time 1786939828.90 total 30.02 last 144.61) - open/close/unlink 4817 (time 1786939838.91 total 40.02 last 141.76) - open/close/unlink 5982 (time 1786939848.91 total 50.03 last 116.43) - open/close/unlink 7260 (time 1786939858.92 total 60.03 last 127.73) - open/close/unlink 9066 (time 1786939868.92 total 70.04 last 180.52) - open/close/unlink 10000 (time 1786939873.69 total 74.81 last 195.78) - open/close/unlink 11824 (time 1786939883.69 total 84.81 last 182.39) - open/close/unlink 13497 (time 1786939893.70 total 94.81 last 167.23) - open/close/unlink 15033 (time 1786939903.70 total 104.81 last 153.57) - open/close/unlink 16827 (time 1786939913.70 total 114.81 last 179.36) - open/close/unlink 18471 (time 1786939923.71 total 124.82 last 164.35) - open/close/unlink 20000 (time 1786939932.20 total 133.32 last 179.89) - open/close/unlink 21823 (time 1786939942.22 total 143.33 last 182.09) - open/close/unlink 23822 (time 1786939952.22 total 153.33 last 199.87) - open/close/unlink 25815 (time 1786939962.22 total 163.33 last 199.24) - open/close/unlink 27720 (time 1786939972.26 total 173.37 last 189.75) - open/close/unlink 29869 (time 1786939982.36 total 183.48 last 212.69) - open/close/unlink 32107 (time 1786939992.37 total 193.48 last 223.72) - open/close/unlink 34345 (time 1786940002.37 total 203.48 last 223.77) - open/close/unlink 36651 (time 1786940012.37 total 213.48 last 230.56) - open/close/unlink 39040 (time 1786940022.37 total 223.48 last 238.87) - open/close/unlink 40000 (time 1786940026.33 total 227.45 last 242.35) - open/close/unlink 42441 (time 1786940036.34 total 237.45 last 243.99) - open/close/unlink 44655 (time 1786940046.34 total 247.45 last 221.29) - open/close/unlink 47011 (time 1786940056.35 total 257.46 last 235.48) - open/close/unlink 49331 (time 1786940066.35 total 267.47 last 231.84) - open/close/unlink 50000 (time 1786940069.57 total 270.68 last 208.04) - open/close/unlink 52327 (time 1786940079.57 total 280.68 last 232.65) - open/close/unlink 54693 (time 1786940089.58 total 290.69 last 236.51) - open/close/unlink 56877 (time 1786940099.58 total 300.69 last 218.32) - open/close/unlink 59020 (time 1786940109.58 total 310.69 last 214.26) - open/close/unlink 60000 (time 1786940113.66 total 314.77 last 240.33) - open/close/unlink 62338 (time 1786940123.66 total 324.77 last 233.80) - open/close/unlink 64658 (time 1786940133.66 total 334.78 last 231.90) total: 64768 open/close/unlink in 336.41 seconds: 192.53 ops/second - open/close/unlink 866 (time 1786939809.04 total 10.01 last 86.52) - open/close/unlink 1958 (time 1786939819.05 total 20.01 last 109.14) - open/close/unlink 3408 (time 1786939829.06 total 30.02 last 144.91) - open/close/unlink 4804 (time 1786939839.06 total 40.03 last 139.49) - open/close/unlink 5965 (time 1786939849.06 total 50.03 last 116.09) - open/close/unlink 7256 (time 1786939859.07 total 60.03 last 129.10) - open/close/unlink 9072 (time 1786939869.08 total 70.04 last 181.40) - open/close/unlink 10000 (time 1786939873.77 total 74.73 last 197.78) - open/close/unlink 11816 (time 1786939883.77 total 84.73 last 181.60) - open/close/unlink 13494 (time 1786939893.77 total 94.74 last 167.71) - open/close/unlink 15025 (time 1786939903.77 total 104.74 last 153.10) - open/close/unlink 16805 (time 1786939913.78 total 114.74 last 177.97) - open/close/unlink 18483 (time 1786939923.78 total 124.74 last 167.72) - open/close/unlink 20000 (time 1786939932.03 total 133.00 last 183.80) - open/close/unlink 21838 (time 1786939942.04 total 143.00 last 183.71) - open/close/unlink 23833 (time 1786939952.04 total 153.01 last 199.45) - open/close/unlink 25797 (time 1786939962.04 total 163.01 last 196.34) - open/close/unlink 27688 (time 1786939972.04 total 173.01 last 189.10) - open/close/unlink 29835 (time 1786939982.05 total 183.01 last 214.62) - open/close/unlink 32082 (time 1786939992.05 total 193.02 last 224.64) - open/close/unlink 34241 (time 1786940002.05 total 203.02 last 215.88) - open/close/unlink 36559 (time 1786940012.05 total 213.02 last 231.72) - open/close/unlink 38859 (time 1786940022.06 total 223.02 last 229.94) - open/close/unlink 40000 (time 1786940026.74 total 227.71 last 243.52) - open/close/unlink 42430 (time 1786940036.74 total 237.71 last 242.95) - open/close/unlink 44639 (time 1786940046.75 total 247.71 last 220.83) - open/close/unlink 47048 (time 1786940056.75 total 257.72 last 240.75) - open/close/unlink 49355 (time 1786940066.76 total 267.72 last 230.61) - open/close/unlink 50000 (time 1786940069.69 total 270.66 last 219.97) - open/close/unlink 52377 (time 1786940079.69 total 280.66 last 237.68) - open/close/unlink 54761 (time 1786940089.69 total 290.66 last 238.34) - open/close/unlink 56942 (time 1786940099.70 total 300.66 last 218.06) - open/close/unlink 59124 (time 1786940109.70 total 310.66 last 218.17) - open/close/unlink 60000 (time 1786940113.24 total 314.21 last 246.88) - open/close/unlink 62263 (time 1786940123.25 total 324.21 last 226.25) - open/close/unlink 64545 (time 1786940133.25 total 334.21 last 228.14) total: 64768 open/close/unlink in 336.91 seconds: 192.24 ops/second - open/close/unlink 867 (time 1786939809.01 total 10.00 last 86.67) - open/close/unlink 1954 (time 1786939819.01 total 20.01 last 108.64) - open/close/unlink 3398 (time 1786939829.02 total 30.02 last 144.31) - open/close/unlink 4814 (time 1786939839.03 total 40.03 last 141.43) - open/close/unlink 5977 (time 1786939849.05 total 50.04 last 116.11) - open/close/unlink 7255 (time 1786939859.05 total 60.05 last 127.74) - open/close/unlink 9099 (time 1786939869.05 total 70.05 last 184.38) - open/close/unlink 10000 (time 1786939873.65 total 74.65 last 195.78) - open/close/unlink 11822 (time 1786939883.66 total 84.65 last 182.19) - open/close/unlink 13525 (time 1786939893.66 total 94.65 last 170.30) - open/close/unlink 15063 (time 1786939903.66 total 104.66 last 153.75) - open/close/unlink 16820 (time 1786939913.66 total 114.66 last 175.66) - open/close/unlink 18464 (time 1786939923.66 total 124.66 last 164.35) - open/close/unlink 20000 (time 1786939932.06 total 133.06 last 182.99) - open/close/unlink 21829 (time 1786939942.06 total 143.06 last 182.86) - open/close/unlink 23820 (time 1786939952.06 total 153.06 last 199.04) - open/close/unlink 25796 (time 1786939962.07 total 163.06 last 197.56) - open/close/unlink 27686 (time 1786939972.07 total 173.07 last 188.94) - open/close/unlink 29842 (time 1786939982.07 total 183.07 last 215.57) - open/close/unlink 32064 (time 1786939992.07 total 193.07 last 222.13) - open/close/unlink 34252 (time 1786940002.08 total 203.07 last 218.73) - open/close/unlink 36577 (time 1786940012.08 total 213.08 last 232.43) - open/close/unlink 38880 (time 1786940022.08 total 223.08 last 230.24) - open/close/unlink 40000 (time 1786940026.56 total 227.55 last 250.37) - open/close/unlink 42449 (time 1786940036.56 total 237.56 last 244.81) - open/close/unlink 44673 (time 1786940046.56 total 247.56 last 222.35) - open/close/unlink 47061 (time 1786940056.56 total 257.56 last 238.77) - open/close/unlink 49345 (time 1786940066.57 total 267.57 last 228.30) - open/close/unlink 50000 (time 1786940069.64 total 270.64 last 212.89) - open/close/unlink 52309 (time 1786940079.65 total 280.64 last 230.87) - open/close/unlink 54655 (time 1786940089.65 total 290.64 last 234.59) - open/close/unlink 56896 (time 1786940099.65 total 300.65 last 224.03) - open/close/unlink 59067 (time 1786940109.65 total 310.65 last 217.05) - open/close/unlink 60000 (time 1786940113.54 total 314.53 last 240.21) - open/close/unlink 62263 (time 1786940123.54 total 324.53 last 226.29) - open/close/unlink 64561 (time 1786940133.54 total 334.54 last 229.73) total: 64768 open/close/unlink in 337.31 seconds: 192.01 ops/second Waiting for MDT destroys to complete oleg338-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg338-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg338-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server stop mds service on oleg338-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server stop mds service on oleg338-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg338-server unloading modules via unload_modules_local on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing unload_modules_local oleg338-server: modules unloaded. pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2