== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 04:52:47 (1785315167) start mds service on oleg611-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg611-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg611-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1 Started lustre-OST0000 oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 275 79717 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 433 49567 1% /mnt/lustre[OST:0] filesystem_summary: 50097 530 49567 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 2106 (time 1785315215.28 total 10.00 last 210.59) - open/close/unlink 3748 (time 1785315225.28 total 20.00 last 164.18) - open/close/unlink 5603 (time 1785315235.28 total 30.01 last 185.44) - open/close/unlink 7612 (time 1785315245.28 total 40.01 last 200.83) - open/close/unlink 9831 (time 1785315255.29 total 50.01 last 221.78) - open/close/unlink 11954 (time 1785315265.29 total 60.01 last 212.30) - open/close/unlink 13914 (time 1785315275.29 total 70.02 last 195.93) - open/close/unlink 15961 (time 1785315285.30 total 80.02 last 204.65) - open/close/unlink 18238 (time 1785315295.30 total 90.02 last 227.62) - open/close/unlink 20000 (time 1785315303.44 total 98.16 last 216.54) - open/close/unlink 22391 (time 1785315313.44 total 108.16 last 239.08) - open/close/unlink 24666 (time 1785315323.44 total 118.16 last 227.50) - open/close/unlink 27000 (time 1785315333.44 total 128.16 last 233.33) - open/close/unlink 29072 (time 1785315343.44 total 138.17 last 207.15) - open/close/unlink 30000 (time 1785315347.84 total 142.56 last 211.27) - open/close/unlink 32000 (time 1785315357.84 total 152.56 last 199.98) - open/close/unlink 34000 (time 1785315367.84 total 162.56 last 199.97) - open/close/unlink 36013 (time 1785315377.84 total 172.56 last 201.29) - open/close/unlink 38277 (time 1785315387.84 total 182.56 last 226.39) - open/close/unlink 40000 (time 1785315395.31 total 190.03 last 230.71) - open/close/unlink 42417 (time 1785315405.31 total 200.04 last 241.62) - open/close/unlink 44721 (time 1785315415.31 total 210.04 last 230.30) - open/close/unlink 47027 (time 1785315425.32 total 220.04 last 230.50) - open/close/unlink 49343 (time 1785315435.33 total 230.05 last 231.40) - open/close/unlink 50000 (time 1785315438.58 total 233.30 last 202.17) - open/close/unlink 52033 (time 1785315448.58 total 243.31 last 203.23) - open/close/unlink 54376 (time 1785315458.58 total 253.31 last 234.28) - open/close/unlink 56746 (time 1785315468.58 total 263.31 last 236.96) - open/close/unlink 59027 (time 1785315478.59 total 273.31 last 228.03) - open/close/unlink 60000 (time 1785315483.42 total 278.14 last 201.36) - open/close/unlink 62374 (time 1785315493.42 total 288.15 last 237.28) - open/close/unlink 64657 (time 1785315503.43 total 298.15 last 228.24) total: 64768 open/close/unlink in 298.60 seconds: 216.91 ops/second - open/close/unlink 2084 (time 1785315215.29 total 10.01 last 208.26) - open/close/unlink 3718 (time 1785315225.29 total 20.01 last 163.35) - open/close/unlink 5592 (time 1785315235.29 total 30.01 last 187.39) - open/close/unlink 7601 (time 1785315245.29 total 40.01 last 200.88) - open/close/unlink 9838 (time 1785315255.30 total 50.02 last 223.61) - open/close/unlink 11946 (time 1785315265.30 total 60.02 last 210.74) - open/close/unlink 13893 (time 1785315275.30 total 70.02 last 194.65) - open/close/unlink 16002 (time 1785315285.30 total 80.02 last 210.88) - open/close/unlink 18252 (time 1785315295.31 total 90.02 last 224.96) - open/close/unlink 20000 (time 1785315303.41 total 98.13 last 215.72) - open/close/unlink 22394 (time 1785315313.41 total 108.13 last 239.39) - open/close/unlink 24671 (time 1785315323.41 total 118.13 last 227.66) - open/close/unlink 26981 (time 1785315333.41 total 128.13 last 230.95) - open/close/unlink 28995 (time 1785315343.41 total 138.13 last 201.37) - open/close/unlink 30000 (time 1785315348.06 total 142.78 last 216.48) - open/close/unlink 32025 (time 1785315358.06 total 152.78 last 202.49) - open/close/unlink 34038 (time 1785315368.06 total 162.78 last 201.26) - open/close/unlink 36013 (time 1785315378.06 total 172.78 last 197.43) - open/close/unlink 38287 (time 1785315388.07 total 182.78 last 227.33) - open/close/unlink 40000 (time 1785315395.62 total 190.34 last 226.76) - open/close/unlink 42417 (time 1785315405.62 total 200.34 last 241.68) - open/close/unlink 44765 (time 1785315415.62 total 210.34 last 234.76) - open/close/unlink 47058 (time 1785315425.62 total 220.34 last 229.29) - open/close/unlink 49354 (time 1785315435.62 total 230.34 last 229.56) - open/close/unlink 50000 (time 1785315438.75 total 233.47 last 206.71) - open/close/unlink 52025 (time 1785315448.75 total 243.47 last 202.47) - open/close/unlink 54351 (time 1785315458.75 total 253.47 last 232.55) - open/close/unlink 56716 (time 1785315468.76 total 263.48 last 236.43) - open/close/unlink 58979 (time 1785315478.76 total 273.48 last 226.24) - open/close/unlink 60000 (time 1785315483.73 total 278.45 last 205.42) - open/close/unlink 62387 (time 1785315493.73 total 288.45 last 238.67) - open/close/unlink 64675 (time 1785315503.74 total 298.45 last 228.70) total: 64768 open/close/unlink in 299.60 seconds: 216.18 ops/second - open/close/unlink 2120 (time 1785315215.02 total 10.00 last 211.97) - open/close/unlink 3755 (time 1785315225.02 total 20.01 last 163.43) - open/close/unlink 5607 (time 1785315235.03 total 30.01 last 185.18) - open/close/unlink 7628 (time 1785315245.03 total 40.01 last 202.06) - open/close/unlink 9828 (time 1785315255.03 total 50.01 last 219.96) - open/close/unlink 11944 (time 1785315265.03 total 60.01 last 211.60) - open/close/unlink 13888 (time 1785315275.03 total 70.01 last 194.33) - open/close/unlink 15931 (time 1785315285.04 total 80.02 last 204.24) - open/close/unlink 18206 (time 1785315295.04 total 90.02 last 227.47) - open/close/unlink 20000 (time 1785315303.43 total 98.42 last 213.64) - open/close/unlink 22339 (time 1785315313.44 total 108.42 last 233.82) - open/close/unlink 24636 (time 1785315323.44 total 118.42 last 229.69) - open/close/unlink 26960 (time 1785315333.44 total 128.42 last 232.35) - open/close/unlink 29014 (time 1785315343.44 total 138.43 last 205.33) - open/close/unlink 30000 (time 1785315348.09 total 143.07 last 212.43) - open/close/unlink 32017 (time 1785315358.09 total 153.07 last 201.64) - open/close/unlink 33983 (time 1785315368.09 total 163.07 last 196.57) - open/close/unlink 35947 (time 1785315378.09 total 173.07 last 196.34) - open/close/unlink 38196 (time 1785315388.09 total 183.07 last 224.89) - open/close/unlink 40000 (time 1785315396.01 total 190.99 last 228.02) - open/close/unlink 42432 (time 1785315406.01 total 200.99 last 243.19) - open/close/unlink 44755 (time 1785315416.01 total 210.99 last 232.21) - open/close/unlink 47060 (time 1785315426.01 total 220.99 last 230.47) - open/close/unlink 49370 (time 1785315436.01 total 230.99 last 230.97) - open/close/unlink 50000 (time 1785315439.09 total 234.07 last 204.63) - open/close/unlink 52047 (time 1785315449.09 total 244.08 last 204.63) - open/close/unlink 54397 (time 1785315459.10 total 254.08 last 234.96) - open/close/unlink 56776 (time 1785315469.10 total 264.08 last 237.88) - open/close/unlink 59026 (time 1785315479.10 total 274.08 last 224.87) - open/close/unlink 60000 (time 1785315483.80 total 278.78 last 207.24) - open/close/unlink 62387 (time 1785315493.80 total 288.79 last 238.66) - open/close/unlink 64681 (time 1785315503.80 total 298.79 last 229.40) total: 64768 open/close/unlink in 300.52 seconds: 215.52 ops/second - open/close/unlink 2096 (time 1785315215.10 total 10.01 last 209.30) - open/close/unlink 3721 (time 1785315225.10 total 20.02 last 162.43) - open/close/unlink 5594 (time 1785315235.11 total 30.03 last 187.17) - open/close/unlink 7606 (time 1785315245.11 total 40.03 last 201.11) - open/close/unlink 9796 (time 1785315255.13 total 50.05 last 218.67) - open/close/unlink 11942 (time 1785315265.13 total 60.05 last 214.55) - open/close/unlink 13880 (time 1785315275.13 total 70.05 last 193.79) - open/close/unlink 15965 (time 1785315285.13 total 80.05 last 208.43) - open/close/unlink 18244 (time 1785315295.14 total 90.05 last 227.87) - open/close/unlink 20000 (time 1785315303.42 total 98.34 last 212.01) - open/close/unlink 22331 (time 1785315313.42 total 108.34 last 233.09) - open/close/unlink 24596 (time 1785315323.42 total 118.34 last 226.42) - open/close/unlink 26924 (time 1785315333.43 total 128.34 last 232.71) - open/close/unlink 28929 (time 1785315343.43 total 138.34 last 200.50) - open/close/unlink 30000 (time 1785315348.59 total 143.51 last 207.48) - open/close/unlink 31992 (time 1785315358.59 total 153.51 last 199.18) - open/close/unlink 34014 (time 1785315368.59 total 163.51 last 202.16) - open/close/unlink 35957 (time 1785315378.59 total 173.51 last 194.28) - open/close/unlink 38242 (time 1785315388.60 total 183.51 last 228.41) - open/close/unlink 40000 (time 1785315396.11 total 191.02 last 234.14) - open/close/unlink 42412 (time 1785315406.11 total 201.02 last 241.19) - open/close/unlink 44729 (time 1785315416.11 total 211.02 last 231.67) - open/close/unlink 47025 (time 1785315426.11 total 221.03 last 229.56) - open/close/unlink 49337 (time 1785315436.11 total 231.03 last 231.12) - open/close/unlink 50000 (time 1785315439.43 total 234.35 last 199.57) - open/close/unlink 52067 (time 1785315449.44 total 244.35 last 206.64) - open/close/unlink 54391 (time 1785315459.44 total 254.36 last 232.33) - open/close/unlink 56758 (time 1785315469.44 total 264.36 last 236.67) - open/close/unlink 58998 (time 1785315479.45 total 274.36 last 223.90) - open/close/unlink 60000 (time 1785315484.10 total 279.02 last 215.16) - open/close/unlink 62356 (time 1785315494.11 total 289.02 last 235.50) - open/close/unlink 64587 (time 1785315504.12 total 299.03 last 222.85) total: 64768 open/close/unlink in 302.74 seconds: 213.94 ops/second - open/close/unlink 2078 (time 1785315215.28 total 10.00 last 207.73) - open/close/unlink 3720 (time 1785315225.28 total 20.01 last 164.15) - open/close/unlink 5592 (time 1785315235.28 total 30.01 last 187.17) - open/close/unlink 7603 (time 1785315245.28 total 40.01 last 201.07) - open/close/unlink 9801 (time 1785315255.29 total 50.02 last 219.64) - open/close/unlink 11953 (time 1785315265.29 total 60.02 last 215.14) - open/close/unlink 13887 (time 1785315275.30 total 70.02 last 193.33) - open/close/unlink 15953 (time 1785315285.30 total 80.02 last 206.58) - open/close/unlink 18235 (time 1785315295.30 total 90.03 last 228.08) - open/close/unlink 20000 (time 1785315303.51 total 98.24 last 214.97) - open/close/unlink 22350 (time 1785315313.52 total 108.24 last 234.96) - open/close/unlink 24598 (time 1785315323.52 total 118.24 last 224.74) - open/close/unlink 26925 (time 1785315333.52 total 128.25 last 232.67) - open/close/unlink 28936 (time 1785315343.52 total 138.25 last 201.04) - open/close/unlink 30000 (time 1785315348.61 total 143.34 last 209.16) - open/close/unlink 31996 (time 1785315358.61 total 153.34 last 199.53) - open/close/unlink 33956 (time 1785315368.61 total 163.34 last 195.99) - open/close/unlink 35921 (time 1785315378.62 total 173.35 last 196.38) - open/close/unlink 38199 (time 1785315388.62 total 183.35 last 227.75) - open/close/unlink 40000 (time 1785315396.46 total 191.19 last 229.66) - open/close/unlink 42415 (time 1785315406.47 total 201.19 last 241.49) - open/close/unlink 44726 (time 1785315416.47 total 211.19 last 231.06) - open/close/unlink 47042 (time 1785315426.47 total 221.19 last 231.58) - open/close/unlink 49314 (time 1785315436.47 total 231.20 last 227.11) - open/close/unlink 50000 (time 1785315439.77 total 234.49 last 208.13) - open/close/unlink 52048 (time 1785315449.77 total 244.50 last 204.76) - open/close/unlink 54370 (time 1785315459.77 total 254.50 last 232.15) - open/close/unlink 56725 (time 1785315469.77 total 264.50 last 235.46) - open/close/unlink 58920 (time 1785315479.78 total 274.50 last 219.47) - open/close/unlink 60000 (time 1785315484.79 total 279.52 last 215.35) - open/close/unlink 62358 (time 1785315494.79 total 289.52 last 235.76) - open/close/unlink 64485 (time 1785315504.80 total 299.53 last 212.47) total: 64768 open/close/unlink in 303.91 seconds: 213.12 ops/second Waiting for MDT destroys to complete oleg611-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg611-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg611-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg611-server stop mds service on oleg611-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg611-server stop mds service on oleg611-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg611-server unloading modules via unload_modules_local on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing unload_modules_local oleg611-server: modules unloaded. pdsh@oleg611-client: oleg611-client: ssh exited with exit code 2 pdsh@oleg611-client: oleg611-server: ssh exited with exit code 2 pdsh@oleg611-client: oleg611-client: ssh exited with exit code 2 pdsh@oleg611-client: oleg611-server: ssh exited with exit code 2