== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:35:29 (1788712529) start mds service on oleg427-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg427-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg427-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 433 49567 1% /mnt/lustre[OST:0] filesystem_summary: 50096 529 49567 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 2097 (time 1788712574.29 total 10.00 last 209.63) - open/close/unlink 4413 (time 1788712584.29 total 20.01 last 231.54) - open/close/unlink 6646 (time 1788712594.29 total 30.01 last 223.26) - open/close/unlink 8854 (time 1788712604.29 total 40.01 last 220.70) - open/close/unlink 10000 (time 1788712609.65 total 45.37 last 213.84) - open/close/unlink 12227 (time 1788712619.66 total 55.37 last 222.68) - open/close/unlink 14395 (time 1788712629.66 total 65.38 last 216.68) - open/close/unlink 16642 (time 1788712639.66 total 75.38 last 224.68) - open/close/unlink 19021 (time 1788712649.67 total 85.38 last 237.76) - open/close/unlink 20000 (time 1788712654.29 total 90.00 last 211.98) - open/close/unlink 22154 (time 1788712664.29 total 100.00 last 215.37) - open/close/unlink 24357 (time 1788712674.29 total 110.01 last 220.27) - open/close/unlink 26516 (time 1788712684.29 total 120.01 last 215.85) - open/close/unlink 28753 (time 1788712694.29 total 130.01 last 223.63) - open/close/unlink 30000 (time 1788712699.91 total 135.63 last 221.87) - open/close/unlink 32141 (time 1788712709.92 total 145.63 last 214.05) - open/close/unlink 34180 (time 1788712719.92 total 155.64 last 203.82) - open/close/unlink 36221 (time 1788712729.92 total 165.64 last 204.05) - open/close/unlink 38145 (time 1788712739.93 total 175.64 last 192.32) - open/close/unlink 40000 (time 1788712748.73 total 184.45 last 210.78) - open/close/unlink 41967 (time 1788712758.73 total 194.45 last 196.67) - open/close/unlink 43705 (time 1788712768.73 total 204.45 last 173.73) - open/close/unlink 45330 (time 1788712778.73 total 214.45 last 162.48) - open/close/unlink 46950 (time 1788712788.74 total 224.46 last 161.95) - open/close/unlink 49072 (time 1788712798.74 total 234.46 last 212.20) - open/close/unlink 50000 (time 1788712802.45 total 238.17 last 250.06) - open/close/unlink 52531 (time 1788712812.45 total 248.17 last 253.03) - open/close/unlink 55128 (time 1788712822.45 total 258.17 last 259.63) - open/close/unlink 57685 (time 1788712832.46 total 268.17 last 255.66) - open/close/unlink 60000 (time 1788712841.17 total 276.88 last 265.79) - open/close/unlink 62716 (time 1788712851.17 total 286.89 last 271.55) total: 64768 open/close/unlink in 294.82 seconds: 219.68 ops/second - open/close/unlink 2114 (time 1788712574.48 total 10.00 last 211.35) - open/close/unlink 4428 (time 1788712584.48 total 20.01 last 231.28) - open/close/unlink 6647 (time 1788712594.48 total 30.01 last 221.90) - open/close/unlink 8866 (time 1788712604.48 total 40.01 last 221.85) - open/close/unlink 10000 (time 1788712609.75 total 45.28 last 215.35) - open/close/unlink 12204 (time 1788712619.75 total 55.28 last 220.34) - open/close/unlink 14353 (time 1788712629.75 total 65.28 last 214.82) - open/close/unlink 16592 (time 1788712639.76 total 75.28 last 223.85) - open/close/unlink 18963 (time 1788712649.76 total 85.29 last 237.06) - open/close/unlink 20000 (time 1788712654.73 total 90.26 last 208.39) - open/close/unlink 22170 (time 1788712664.74 total 100.26 last 216.95) - open/close/unlink 24386 (time 1788712674.74 total 110.27 last 221.56) - open/close/unlink 26532 (time 1788712684.74 total 120.27 last 214.59) - open/close/unlink 28770 (time 1788712694.74 total 130.27 last 223.78) - open/close/unlink 30000 (time 1788712700.37 total 135.90 last 218.45) - open/close/unlink 32147 (time 1788712710.37 total 145.90 last 214.68) - open/close/unlink 34195 (time 1788712720.37 total 155.90 last 204.77) - open/close/unlink 36275 (time 1788712730.38 total 165.90 last 207.96) - open/close/unlink 38159 (time 1788712740.38 total 175.90 last 188.37) - open/close/unlink 40000 (time 1788712749.06 total 184.59 last 211.91) - open/close/unlink 41991 (time 1788712759.07 total 194.60 last 199.03) - open/close/unlink 43727 (time 1788712769.07 total 204.60 last 173.58) - open/close/unlink 45337 (time 1788712779.07 total 214.60 last 160.97) - open/close/unlink 46969 (time 1788712789.07 total 224.60 last 163.18) - open/close/unlink 49148 (time 1788712799.07 total 234.60 last 217.88) - open/close/unlink 50000 (time 1788712802.42 total 237.95 last 254.35) - open/close/unlink 52527 (time 1788712812.42 total 247.95 last 252.70) - open/close/unlink 55055 (time 1788712822.42 total 257.95 last 252.78) - open/close/unlink 57667 (time 1788712832.42 total 267.95 last 261.19) - open/close/unlink 60000 (time 1788712841.13 total 276.65 last 268.12) - open/close/unlink 62676 (time 1788712851.13 total 286.66 last 267.53) total: 64768 open/close/unlink in 294.90 seconds: 219.62 ops/second - open/close/unlink 2106 (time 1788712574.50 total 10.00 last 210.51) - open/close/unlink 4427 (time 1788712584.50 total 20.01 last 232.06) - open/close/unlink 6627 (time 1788712594.50 total 30.01 last 219.91) - open/close/unlink 8831 (time 1788712604.51 total 40.02 last 220.29) - open/close/unlink 10000 (time 1788712609.91 total 45.42 last 216.44) - open/close/unlink 12190 (time 1788712619.91 total 55.42 last 218.93) - open/close/unlink 14369 (time 1788712629.92 total 65.42 last 217.82) - open/close/unlink 16604 (time 1788712639.92 total 75.43 last 223.37) - open/close/unlink 18969 (time 1788712649.92 total 85.43 last 236.48) - open/close/unlink 20000 (time 1788712654.79 total 90.30 last 211.68) - open/close/unlink 22190 (time 1788712664.80 total 100.30 last 218.96) - open/close/unlink 24345 (time 1788712674.80 total 110.30 last 215.50) - open/close/unlink 26489 (time 1788712684.80 total 120.30 last 214.36) - open/close/unlink 28704 (time 1788712694.82 total 130.33 last 220.91) - open/close/unlink 30000 (time 1788712700.58 total 136.09 last 225.16) - open/close/unlink 32096 (time 1788712710.58 total 146.09 last 209.57) - open/close/unlink 34115 (time 1788712720.58 total 156.09 last 201.88) - open/close/unlink 36185 (time 1788712730.59 total 166.09 last 206.94) - open/close/unlink 38068 (time 1788712740.59 total 176.10 last 188.21) - open/close/unlink 40000 (time 1788712749.96 total 185.46 last 206.31) - open/close/unlink 42010 (time 1788712759.96 total 195.46 last 200.97) - open/close/unlink 43740 (time 1788712769.96 total 205.47 last 172.93) - open/close/unlink 45317 (time 1788712779.97 total 215.47 last 157.63) - open/close/unlink 47019 (time 1788712789.97 total 225.48 last 170.15) - open/close/unlink 49216 (time 1788712799.97 total 235.48 last 219.64) - open/close/unlink 50000 (time 1788712803.07 total 238.57 last 253.22) - open/close/unlink 52521 (time 1788712813.07 total 248.58 last 252.03) - open/close/unlink 55094 (time 1788712823.07 total 258.58 last 257.27) - open/close/unlink 57683 (time 1788712833.07 total 268.58 last 258.86) - open/close/unlink 60000 (time 1788712841.63 total 277.14 last 270.62) - open/close/unlink 62728 (time 1788712851.64 total 287.15 last 272.65) total: 64768 open/close/unlink in 296.29 seconds: 218.59 ops/second - open/close/unlink 2112 (time 1788712574.47 total 10.00 last 211.12) - open/close/unlink 4434 (time 1788712584.48 total 20.01 last 232.04) - open/close/unlink 6666 (time 1788712594.48 total 30.01 last 223.11) - open/close/unlink 8856 (time 1788712604.48 total 40.02 last 218.95) - open/close/unlink 10000 (time 1788712609.85 total 45.38 last 213.39) - open/close/unlink 12203 (time 1788712619.85 total 55.38 last 220.26) - open/close/unlink 14342 (time 1788712629.85 total 65.38 last 213.84) - open/close/unlink 16587 (time 1788712639.85 total 75.38 last 224.48) - open/close/unlink 18951 (time 1788712649.85 total 85.39 last 236.36) - open/close/unlink 20000 (time 1788712654.94 total 90.48 last 205.99) - open/close/unlink 22187 (time 1788712664.95 total 100.48 last 218.69) - open/close/unlink 24385 (time 1788712674.95 total 110.48 last 219.73) - open/close/unlink 26541 (time 1788712684.95 total 120.48 last 215.56) - open/close/unlink 28760 (time 1788712694.95 total 130.48 last 221.86) - open/close/unlink 30000 (time 1788712700.66 total 136.20 last 217.15) - open/close/unlink 32158 (time 1788712710.66 total 146.20 last 215.75) - open/close/unlink 34189 (time 1788712720.67 total 156.20 last 203.05) - open/close/unlink 36261 (time 1788712730.67 total 166.20 last 207.20) - open/close/unlink 38125 (time 1788712740.67 total 176.20 last 186.39) - open/close/unlink 40000 (time 1788712749.59 total 185.13 last 210.08) - open/close/unlink 41986 (time 1788712759.60 total 195.14 last 198.40) - open/close/unlink 43720 (time 1788712769.61 total 205.14 last 173.28) - open/close/unlink 45323 (time 1788712779.61 total 215.15 last 160.22) - open/close/unlink 47010 (time 1788712789.62 total 225.15 last 168.60) - open/close/unlink 49217 (time 1788712799.62 total 235.15 last 220.69) - open/close/unlink 50000 (time 1788712802.81 total 238.34 last 245.65) - open/close/unlink 52535 (time 1788712812.81 total 248.35 last 253.40) - open/close/unlink 55125 (time 1788712822.82 total 258.35 last 258.92) - open/close/unlink 57671 (time 1788712832.84 total 268.38 last 253.91) - open/close/unlink 60000 (time 1788712841.41 total 276.95 last 271.77) - open/close/unlink 62666 (time 1788712851.41 total 286.95 last 266.60) total: 64768 open/close/unlink in 296.35 seconds: 218.55 ops/second - open/close/unlink 2089 (time 1788712574.50 total 10.00 last 208.87) - open/close/unlink 4425 (time 1788712584.51 total 20.00 last 233.53) - open/close/unlink 6643 (time 1788712594.51 total 30.01 last 221.66) - open/close/unlink 8838 (time 1788712604.52 total 40.02 last 219.37) - open/close/unlink 10000 (time 1788712609.82 total 45.31 last 219.37) - open/close/unlink 12209 (time 1788712619.82 total 55.32 last 220.86) - open/close/unlink 14395 (time 1788712629.82 total 65.32 last 218.54) - open/close/unlink 16646 (time 1788712639.82 total 75.32 last 225.04) - open/close/unlink 18992 (time 1788712649.83 total 85.33 last 234.50) - open/close/unlink 20000 (time 1788712654.79 total 90.29 last 203.16) - open/close/unlink 22191 (time 1788712664.79 total 100.29 last 219.04) - open/close/unlink 24404 (time 1788712674.79 total 110.29 last 221.26) - open/close/unlink 26539 (time 1788712684.80 total 120.29 last 213.44) - open/close/unlink 28760 (time 1788712694.82 total 130.32 last 221.51) - open/close/unlink 30000 (time 1788712700.53 total 136.03 last 217.33) - open/close/unlink 32101 (time 1788712710.53 total 146.03 last 209.99) - open/close/unlink 34117 (time 1788712720.54 total 156.04 last 201.54) - open/close/unlink 36182 (time 1788712730.54 total 166.04 last 206.40) - open/close/unlink 38072 (time 1788712740.54 total 176.04 last 188.98) - open/close/unlink 40000 (time 1788712749.82 total 185.32 last 207.77) - open/close/unlink 41987 (time 1788712759.82 total 195.32 last 198.69) - open/close/unlink 43714 (time 1788712769.83 total 205.33 last 172.62) - open/close/unlink 45297 (time 1788712779.84 total 215.33 last 158.17) - open/close/unlink 46976 (time 1788712789.84 total 225.34 last 167.88) - open/close/unlink 49192 (time 1788712799.84 total 235.34 last 221.54) - open/close/unlink 50000 (time 1788712803.07 total 238.57 last 249.89) - open/close/unlink 52511 (time 1788712813.07 total 248.57 last 251.09) - open/close/unlink 55102 (time 1788712823.08 total 258.57 last 259.04) - open/close/unlink 57684 (time 1788712833.08 total 268.58 last 258.14) - open/close/unlink 60000 (time 1788712841.74 total 277.24 last 267.35) - open/close/unlink 62711 (time 1788712851.74 total 287.24 last 271.04) total: 64768 open/close/unlink in 297.04 seconds: 218.04 ops/second Waiting for MDT destroys to complete oleg427-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local oleg427-server: kmemleak disabled modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2