== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:21:52 (1778347312) start mds service on oleg417-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg417-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 432 49568 1% /mnt/lustre[OST:0] filesystem_summary: 50097 529 49568 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 745 (time 1778347362.51 total 10.01 last 74.44) - open/close/unlink 1676 (time 1778347372.51 total 20.01 last 93.06) - open/close/unlink 3321 (time 1778347382.52 total 30.02 last 164.35) - open/close/unlink 5501 (time 1778347392.52 total 40.02 last 217.97) - open/close/unlink 7349 (time 1778347402.53 total 50.03 last 184.72) - open/close/unlink 9098 (time 1778347412.53 total 60.03 last 174.84) - open/close/unlink 10000 (time 1778347418.29 total 65.79 last 156.70) - open/close/unlink 11795 (time 1778347428.29 total 75.79 last 179.46) - open/close/unlink 14118 (time 1778347438.29 total 85.79 last 232.27) - open/close/unlink 16707 (time 1778347448.29 total 95.79 last 258.84) - open/close/unlink 19230 (time 1778347458.29 total 105.79 last 252.27) - open/close/unlink 20000 (time 1778347461.63 total 109.13 last 230.67) - open/close/unlink 22505 (time 1778347471.64 total 119.14 last 250.35) - open/close/unlink 25022 (time 1778347481.64 total 129.14 last 251.58) - open/close/unlink 27516 (time 1778347491.64 total 139.14 last 249.39) - open/close/unlink 29917 (time 1778347501.65 total 149.15 last 239.99) - open/close/unlink 32224 (time 1778347511.65 total 159.15 last 230.64) - open/close/unlink 34314 (time 1778347521.65 total 169.15 last 208.96) - open/close/unlink 36486 (time 1778347531.65 total 179.15 last 217.18) - open/close/unlink 38906 (time 1778347541.65 total 189.15 last 241.96) - open/close/unlink 40000 (time 1778347546.01 total 193.51 last 251.35) - open/close/unlink 42181 (time 1778347556.01 total 203.51 last 218.06) - open/close/unlink 44420 (time 1778347566.01 total 213.51 last 223.81) - open/close/unlink 46689 (time 1778347576.01 total 223.51 last 226.86) - open/close/unlink 48668 (time 1778347586.02 total 233.52 last 197.83) - open/close/unlink 50000 (time 1778347592.86 total 240.36 last 194.66) - open/close/unlink 51927 (time 1778347602.86 total 250.36 last 192.69) - open/close/unlink 54045 (time 1778347612.86 total 260.36 last 211.75) - open/close/unlink 56568 (time 1778347622.87 total 270.36 last 252.27) - open/close/unlink 58594 (time 1778347632.87 total 280.37 last 202.58) - open/close/unlink 60000 (time 1778347639.74 total 287.24 last 204.52) - open/close/unlink 62028 (time 1778347649.75 total 297.25 last 202.69) - open/close/unlink 64085 (time 1778347659.75 total 307.25 last 205.62) total: 64768 open/close/unlink in 309.70 seconds: 209.13 ops/second - open/close/unlink 744 (time 1778347362.47 total 10.00 last 74.38) - open/close/unlink 1661 (time 1778347372.48 total 20.01 last 91.64) - open/close/unlink 3295 (time 1778347382.48 total 30.01 last 163.38) - open/close/unlink 5483 (time 1778347392.48 total 40.01 last 218.76) - open/close/unlink 7325 (time 1778347402.48 total 50.01 last 184.16) - open/close/unlink 9084 (time 1778347412.50 total 60.03 last 175.61) - open/close/unlink 10000 (time 1778347418.25 total 65.78 last 159.40) - open/close/unlink 11786 (time 1778347428.25 total 75.78 last 178.58) - open/close/unlink 14113 (time 1778347438.25 total 85.78 last 232.69) - open/close/unlink 16714 (time 1778347448.25 total 95.78 last 260.05) - open/close/unlink 19211 (time 1778347458.25 total 105.78 last 249.63) - open/close/unlink 20000 (time 1778347461.66 total 109.19 last 231.91) - open/close/unlink 22497 (time 1778347471.66 total 119.19 last 249.64) - open/close/unlink 25019 (time 1778347481.66 total 129.19 last 252.09) - open/close/unlink 27507 (time 1778347491.67 total 139.20 last 248.65) - open/close/unlink 29915 (time 1778347501.67 total 149.20 last 240.77) - open/close/unlink 32198 (time 1778347511.67 total 159.20 last 228.29) - open/close/unlink 34279 (time 1778347521.67 total 169.20 last 208.09) - open/close/unlink 36446 (time 1778347531.67 total 179.20 last 216.65) - open/close/unlink 38904 (time 1778347541.68 total 189.21 last 245.70) - open/close/unlink 40000 (time 1778347546.04 total 193.57 last 251.03) - open/close/unlink 42135 (time 1778347556.04 total 203.57 last 213.49) - open/close/unlink 44395 (time 1778347566.05 total 213.58 last 225.98) - open/close/unlink 46660 (time 1778347576.05 total 223.58 last 226.48) - open/close/unlink 48627 (time 1778347586.05 total 233.58 last 196.66) - open/close/unlink 50000 (time 1778347593.09 total 240.62 last 195.10) - open/close/unlink 51898 (time 1778347603.09 total 250.62 last 189.71) - open/close/unlink 54013 (time 1778347613.09 total 260.62 last 211.49) - open/close/unlink 56518 (time 1778347623.09 total 270.62 last 250.49) - open/close/unlink 58588 (time 1778347633.09 total 280.62 last 206.94) - open/close/unlink 60000 (time 1778347640.21 total 287.74 last 198.55) - open/close/unlink 62017 (time 1778347650.21 total 297.74 last 201.67) - open/close/unlink 64137 (time 1778347660.21 total 307.74 last 211.95) total: 64768 open/close/unlink in 311.51 seconds: 207.91 ops/second - open/close/unlink 752 (time 1778347362.33 total 10.01 last 75.11) - open/close/unlink 1681 (time 1778347372.34 total 20.02 last 92.83) - open/close/unlink 3304 (time 1778347382.34 total 30.02 last 162.30) - open/close/unlink 5491 (time 1778347392.35 total 40.02 last 218.61) - open/close/unlink 7345 (time 1778347402.35 total 50.03 last 185.36) - open/close/unlink 9094 (time 1778347412.35 total 60.03 last 174.89) - open/close/unlink 10000 (time 1778347418.14 total 65.81 last 156.55) - open/close/unlink 11798 (time 1778347428.14 total 75.82 last 179.75) - open/close/unlink 14108 (time 1778347438.14 total 85.82 last 230.99) - open/close/unlink 16688 (time 1778347448.14 total 95.82 last 257.99) - open/close/unlink 19173 (time 1778347458.14 total 105.82 last 248.47) - open/close/unlink 20000 (time 1778347461.64 total 109.32 last 236.24) - open/close/unlink 22504 (time 1778347471.65 total 119.33 last 250.28) - open/close/unlink 25025 (time 1778347481.65 total 129.33 last 252.09) - open/close/unlink 27515 (time 1778347491.65 total 139.33 last 248.90) - open/close/unlink 29924 (time 1778347501.65 total 149.33 last 240.89) - open/close/unlink 32229 (time 1778347511.65 total 159.33 last 230.46) - open/close/unlink 34318 (time 1778347521.66 total 169.34 last 208.81) - open/close/unlink 36495 (time 1778347531.66 total 179.34 last 217.62) - open/close/unlink 38937 (time 1778347541.66 total 189.34 last 244.16) - open/close/unlink 40000 (time 1778347545.90 total 193.58 last 250.94) - open/close/unlink 42145 (time 1778347555.90 total 203.58 last 214.46) - open/close/unlink 44380 (time 1778347565.90 total 213.58 last 223.48) - open/close/unlink 46646 (time 1778347575.90 total 223.58 last 226.59) - open/close/unlink 48627 (time 1778347585.90 total 233.58 last 198.06) - open/close/unlink 50000 (time 1778347593.20 total 240.88 last 188.23) - open/close/unlink 51921 (time 1778347603.20 total 250.88 last 192.07) - open/close/unlink 54030 (time 1778347613.20 total 260.88 last 210.89) - open/close/unlink 56507 (time 1778347623.20 total 270.88 last 247.66) - open/close/unlink 58560 (time 1778347633.20 total 280.88 last 205.25) - open/close/unlink 60000 (time 1778347640.40 total 288.08 last 200.13) - open/close/unlink 62018 (time 1778347650.40 total 298.08 last 201.78) - open/close/unlink 64164 (time 1778347660.40 total 308.08 last 214.55) total: 64768 open/close/unlink in 312.58 seconds: 207.21 ops/second - open/close/unlink 745 (time 1778347362.46 total 10.01 last 74.39) - open/close/unlink 1662 (time 1778347372.46 total 20.02 last 91.68) - open/close/unlink 3295 (time 1778347382.47 total 30.02 last 163.27) - open/close/unlink 5468 (time 1778347392.47 total 40.02 last 217.18) - open/close/unlink 7324 (time 1778347402.48 total 50.03 last 185.50) - open/close/unlink 9073 (time 1778347412.48 total 60.03 last 174.87) - open/close/unlink 10000 (time 1778347418.34 total 65.89 last 158.08) - open/close/unlink 11800 (time 1778347428.34 total 75.90 last 179.98) - open/close/unlink 14138 (time 1778347438.35 total 85.90 last 233.74) - open/close/unlink 16702 (time 1778347448.35 total 95.90 last 256.38) - open/close/unlink 19216 (time 1778347458.35 total 105.90 last 251.33) - open/close/unlink 20000 (time 1778347461.69 total 109.24 last 234.76) - open/close/unlink 22508 (time 1778347471.69 total 119.24 last 250.79) - open/close/unlink 25033 (time 1778347481.69 total 129.24 last 252.46) - open/close/unlink 27508 (time 1778347491.69 total 139.24 last 247.48) - open/close/unlink 29903 (time 1778347501.69 total 149.25 last 239.46) - open/close/unlink 32206 (time 1778347511.70 total 159.25 last 230.23) - open/close/unlink 34291 (time 1778347521.70 total 169.25 last 208.49) - open/close/unlink 36444 (time 1778347531.70 total 179.25 last 215.24) - open/close/unlink 38873 (time 1778347541.70 total 189.25 last 242.88) - open/close/unlink 40000 (time 1778347546.14 total 193.70 last 253.63) - open/close/unlink 42158 (time 1778347556.15 total 203.70 last 215.76) - open/close/unlink 44377 (time 1778347566.15 total 213.70 last 221.83) - open/close/unlink 46641 (time 1778347576.15 total 223.70 last 226.37) - open/close/unlink 48598 (time 1778347586.15 total 233.71 last 195.64) - open/close/unlink 50000 (time 1778347593.56 total 241.11 last 189.38) - open/close/unlink 51921 (time 1778347603.56 total 251.11 last 192.07) - open/close/unlink 54032 (time 1778347613.56 total 261.11 last 211.04) - open/close/unlink 56492 (time 1778347623.56 total 271.11 last 246.00) - open/close/unlink 58520 (time 1778347633.56 total 281.12 last 202.72) - open/close/unlink 60000 (time 1778347640.97 total 288.52 last 199.80) - open/close/unlink 62006 (time 1778347650.97 total 298.52 last 200.60) - open/close/unlink 64176 (time 1778347660.97 total 308.53 last 216.96) total: 64768 open/close/unlink in 313.79 seconds: 206.41 ops/second - open/close/unlink 742 (time 1778347362.51 total 10.01 last 74.12) - open/close/unlink 1678 (time 1778347372.52 total 20.02 last 93.52) - open/close/unlink 3308 (time 1778347382.53 total 30.02 last 162.92) - open/close/unlink 5496 (time 1778347392.53 total 40.03 last 218.73) - open/close/unlink 7344 (time 1778347402.53 total 50.03 last 184.73) - open/close/unlink 9096 (time 1778347412.54 total 60.03 last 175.16) - open/close/unlink 10000 (time 1778347418.27 total 65.76 last 157.79) - open/close/unlink 11798 (time 1778347428.27 total 75.76 last 179.76) - open/close/unlink 14132 (time 1778347438.27 total 85.77 last 233.34) - open/close/unlink 16690 (time 1778347448.27 total 95.77 last 255.76) - open/close/unlink 19161 (time 1778347458.27 total 105.77 last 247.04) - open/close/unlink 20000 (time 1778347461.79 total 109.29 last 238.70) - open/close/unlink 22467 (time 1778347471.79 total 119.29 last 246.65) - open/close/unlink 25004 (time 1778347481.79 total 129.29 last 253.64) - open/close/unlink 27476 (time 1778347491.80 total 139.29 last 247.17) - open/close/unlink 29861 (time 1778347501.80 total 149.30 last 238.39) - open/close/unlink 32155 (time 1778347511.80 total 159.30 last 229.32) - open/close/unlink 34248 (time 1778347521.80 total 169.30 last 209.28) - open/close/unlink 36433 (time 1778347531.80 total 179.30 last 218.49) - open/close/unlink 38862 (time 1778347541.81 total 189.30 last 242.88) - open/close/unlink 40000 (time 1778347546.29 total 193.79 last 253.76) - open/close/unlink 42130 (time 1778347556.29 total 203.79 last 212.93) - open/close/unlink 44328 (time 1778347566.30 total 213.79 last 219.75) - open/close/unlink 46602 (time 1778347576.30 total 223.79 last 227.37) - open/close/unlink 48577 (time 1778347586.30 total 233.80 last 197.44) - open/close/unlink 50000 (time 1778347593.93 total 241.43 last 186.37) - open/close/unlink 51941 (time 1778347603.94 total 251.43 last 194.03) - open/close/unlink 54057 (time 1778347613.94 total 261.44 last 211.58) - open/close/unlink 56495 (time 1778347623.94 total 271.44 last 243.70) - open/close/unlink 58520 (time 1778347633.94 total 281.44 last 202.48) - open/close/unlink 60000 (time 1778347641.26 total 288.76 last 202.27) - open/close/unlink 62036 (time 1778347651.26 total 298.76 last 203.56) - open/close/unlink 64234 (time 1778347661.26 total 308.76 last 219.79) total: 64768 open/close/unlink in 314.10 seconds: 206.20 ops/second Waiting for MDT destroys to complete oleg417-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg417-server unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local oleg417-server: modules unloaded. pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2