== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:15:34 (1734257734) Stopping clients: oleg324-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg324-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 oleg324-server: oleg324-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: Force libcfs to create 2 CPU partitions oleg324-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg324-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg324-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg324-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2241 (time 1734257791.90 total 10.00 last 224.06) - open/close 4003 (time 1734257801.91 total 20.01 last 176.14) - open/close 5707 (time 1734257811.91 total 30.01 last 170.27) - open/close 7477 (time 1734257821.92 total 40.02 last 176.95) - open/close 9329 (time 1734257831.92 total 50.02 last 185.14) - open/close 10000 (time 1734257835.52 total 53.62 last 186.23) - open/close 12204 (time 1734257845.53 total 63.63 last 220.31) - open/close 14874 (time 1734257855.53 total 73.63 last 266.94) - open/close 16991 (time 1734257865.53 total 83.63 last 211.70) - open/close 18770 (time 1734257875.53 total 93.63 last 177.88) - open/close 20000 (time 1734257882.59 total 100.69 last 174.15) - open/close 21877 (time 1734257892.60 total 110.70 last 187.56) - open/close 23655 (time 1734257902.60 total 120.70 last 177.76) - open/close 25522 (time 1734257912.61 total 130.71 last 186.62) - open/close 27113 (time 1734257922.61 total 140.71 last 159.07) - open/close 28742 (time 1734257932.61 total 150.71 last 162.84) - open/close 30000 (time 1734257941.17 total 159.27 last 147.03) - open/close 31612 (time 1734257951.17 total 169.27 last 161.15) - open/close 33332 (time 1734257961.17 total 179.27 last 171.96) - open/close 35096 (time 1734257971.17 total 189.27 last 176.38) - open/close 36838 (time 1734257981.18 total 199.28 last 174.15) - open/close 38661 (time 1734257991.18 total 209.28 last 182.24) - open/close 40000 (time 1734257998.68 total 216.78 last 178.64) - open/close 41628 (time 1734258008.68 total 226.78 last 162.73) - open/close 43377 (time 1734258018.68 total 236.78 last 174.85) - open/close 45065 (time 1734258028.69 total 246.79 last 168.62) - open/close 46506 (time 1734258038.70 total 256.80 last 144.03) - open/close 48077 (time 1734258048.70 total 266.80 last 157.07) - open/close 49648 (time 1734258058.70 total 276.80 last 157.06) - open/close 50000 (time 1734258060.89 total 278.99 last 160.88) - open/close 51673 (time 1734258070.90 total 289.00 last 167.18) - open/close 53150 (time 1734258080.90 total 299.00 last 147.68) - open/close 54695 (time 1734258090.90 total 309.00 last 154.47) - open/close 56057 (time 1734258100.91 total 319.01 last 136.15) - open/close 57423 (time 1734258110.91 total 329.01 last 136.53) - open/close 58776 (time 1734258120.91 total 339.01 last 135.26) - open/close 60000 (time 1734258129.84 total 347.94 last 137.17) - open/close 61418 (time 1734258139.84 total 357.94 last 141.76) - open/close 62893 (time 1734258149.84 total 367.94 last 147.48) - open/close 64239 (time 1734258159.84 total 377.94 last 134.56) total: 64768 open/close in 381.81 seconds: 169.63 ops/second - /unlink 2402 (time 1734258175.44 total 10.00 last 240.09) - /unlink 4608 (time 1734258185.44 total 20.01 last 220.52) - /unlink 7011 (time 1734258195.44 total 30.01 last 240.28) - /unlink 9400 (time 1734258205.45 total 40.01 last 238.85) - /unlink 10000 (time 1734258207.86 total 42.42 last 248.95) - /unlink 12565 (time 1734258217.86 total 52.42 last 256.47) - /unlink 15355 (time 1734258227.86 total 62.42 last 278.96) - /unlink 17953 (time 1734258237.86 total 72.42 last 259.80) - /unlink 20000 (time 1734258244.80 total 79.36 last 295.05) - /unlink 22543 (time 1734258254.80 total 89.36 last 254.24) - /unlink 24906 (time 1734258264.80 total 99.37 last 236.16) - /unlink 27180 (time 1734258274.81 total 109.37 last 227.32) - /unlink 29759 (time 1734258284.81 total 119.37 last 257.89) - /unlink 32570 (time 1734258294.81 total 129.38 last 281.05) - /unlink 35537 (time 1734258304.81 total 139.38 last 296.61) - /unlink 39131 (time 1734258314.82 total 149.38 last 359.31) - /unlink 40000 (time 1734258317.22 total 151.78 last 362.24) - /unlink 43286 (time 1734258327.22 total 161.78 last 328.57) - /unlink 46536 (time 1734258337.22 total 171.78 last 324.99) - /unlink 49328 (time 1734258347.22 total 181.78 last 279.13) - /unlink 50000 (time 1734258349.60 total 184.17 last 281.78) - /unlink 53173 (time 1734258359.60 total 194.17 last 317.27) - /unlink 56127 (time 1734258369.61 total 204.17 last 295.36) - /unlink 58837 (time 1734258379.61 total 214.18 last 270.90) - /unlink 60000 (time 1734258384.18 total 218.74 last 254.62) - /unlink 62869 (time 1734258394.18 total 228.74 last 286.87) total: 64768 /unlink in 234.85 seconds: 275.79 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1785 (time 1734258428.40 total 10.00 last 178.49) - open/close 3310 (time 1734258438.40 total 20.00 last 152.49) - open/close 4890 (time 1734258448.40 total 30.00 last 157.99) - open/close 6590 (time 1734258458.40 total 40.00 last 169.97) - open/close 8331 (time 1734258468.40 total 50.00 last 174.09) - open/close 10000 (time 1734258477.70 total 59.31 last 179.37) - open/close 11718 (time 1734258487.71 total 69.31 last 171.74) - open/close 13553 (time 1734258497.71 total 79.31 last 183.49) - open/close 15701 (time 1734258507.71 total 89.31 last 214.80) - open/close 17731 (time 1734258517.71 total 99.31 last 202.99) - open/close 19622 (time 1734258527.71 total 109.32 last 189.01) - open/close 20000 (time 1734258529.77 total 111.38 last 183.51) - open/close 22149 (time 1734258539.78 total 121.38 last 214.77) - open/close 24427 (time 1734258549.78 total 131.39 last 227.76) - open/close 26365 (time 1734258559.78 total 141.39 last 193.75) - open/close 28555 (time 1734258569.79 total 151.39 last 218.91) - open/close 30000 (time 1734258576.78 total 158.38 last 206.71) - open/close 31784 (time 1734258586.78 total 168.39 last 178.35) - open/close 33498 (time 1734258596.78 total 178.39 last 171.38) - open/close 35225 (time 1734258606.78 total 188.39 last 172.66) - open/close 36927 (time 1734258616.79 total 198.39 last 170.11) - open/close 38841 (time 1734258626.80 total 208.40 last 191.31) - open/close 40000 (time 1734258632.63 total 214.24 last 198.58) - open/close 41675 (time 1734258642.63 total 224.24 last 167.44) - open/close 43244 (time 1734258652.64 total 234.24 last 156.86) - open/close 44906 (time 1734258662.64 total 244.24 last 166.17) - open/close 46453 (time 1734258672.64 total 254.24 last 154.69) - open/close 47953 (time 1734258682.64 total 264.25 last 149.94) - open/close 49611 (time 1734258692.65 total 274.25 last 165.73) - open/close 50000 (time 1734258695.17 total 276.78 last 154.19) - open/close 51541 (time 1734258705.17 total 286.78 last 154.05) - open/close 53075 (time 1734258715.18 total 296.78 last 153.39) - open/close 54724 (time 1734258725.18 total 306.78 last 164.84) - open/close 56490 (time 1734258735.18 total 316.78 last 176.58) - open/close 58264 (time 1734258745.19 total 326.79 last 177.31) - open/close 60000 (time 1734258754.54 total 336.15 last 185.49) - open/close 61737 (time 1734258764.55 total 346.15 last 173.68) - open/close 63701 (time 1734258774.55 total 356.15 last 196.34) total: 64768 open/close in 362.72 seconds: 178.56 ops/second - /unlink 2827 (time 1734258792.55 total 10.00 last 282.58) - /unlink 5593 (time 1734258802.55 total 20.01 last 276.56) - /unlink 8567 (time 1734258812.55 total 30.01 last 297.35) - /unlink 10000 (time 1734258816.36 total 33.82 last 376.11) - /unlink 13159 (time 1734258826.37 total 43.82 last 315.79) - /unlink 16830 (time 1734258836.37 total 53.82 last 367.08) - /unlink 20000 (time 1734258846.27 total 63.72 last 320.10) - /unlink 23444 (time 1734258856.27 total 73.73 last 344.35) - /unlink 26931 (time 1734258866.27 total 83.73 last 348.66) - /unlink 30000 (time 1734258876.22 total 93.67 last 308.53) - /unlink 32771 (time 1734258886.22 total 103.68 last 277.06) - /unlink 35526 (time 1734258896.22 total 113.68 last 275.43) - /unlink 38156 (time 1734258906.23 total 123.68 last 262.94) - /unlink 40000 (time 1734258913.45 total 130.91 last 255.24) - /unlink 42477 (time 1734258923.45 total 140.91 last 247.68) - /unlink 45096 (time 1734258933.45 total 150.91 last 261.88) - /unlink 47851 (time 1734258943.45 total 160.91 last 275.49) - /unlink 50000 (time 1734258950.88 total 168.33 last 289.47) - /unlink 52756 (time 1734258960.88 total 178.33 last 275.54) - /unlink 55684 (time 1734258970.88 total 188.33 last 292.74) - /unlink 58581 (time 1734258980.88 total 198.34 last 289.65) - /unlink 60000 (time 1734258985.32 total 202.78 last 319.52) - /unlink 63375 (time 1734258995.33 total 212.78 last 337.44) total: 64768 /unlink in 216.92 seconds: 298.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1756 (time 1734259028.51 total 10.00 last 175.58) - open/close 3483 (time 1734259038.51 total 20.00 last 172.68) - open/close 5169 (time 1734259048.52 total 30.01 last 168.53) - open/close 6995 (time 1734259058.52 total 40.01 last 182.54) - open/close 8651 (time 1734259068.52 total 50.02 last 165.51) - open/close 10000 (time 1734259077.16 total 58.65 last 156.24) - open/close 11612 (time 1734259087.16 total 68.65 last 161.16) - open/close 13267 (time 1734259097.16 total 78.65 last 165.49) - open/close 14795 (time 1734259107.17 total 88.66 last 152.73) - open/close 16595 (time 1734259117.17 total 98.66 last 179.96) - open/close 18537 (time 1734259127.17 total 108.66 last 194.15) - open/close 20000 (time 1734259135.94 total 117.43 last 166.90) - open/close 21618 (time 1734259145.94 total 127.43 last 161.74) - open/close 23328 (time 1734259155.94 total 137.43 last 170.95) - open/close 25129 (time 1734259165.94 total 147.44 last 180.09) - open/close 27108 (time 1734259175.95 total 157.44 last 197.82) - open/close 28938 (time 1734259185.96 total 167.45 last 182.86) - open/close 30000 (time 1734259192.66 total 174.15 last 158.43) - open/close 31765 (time 1734259202.66 total 184.15 last 176.45) - open/close 33390 (time 1734259212.66 total 194.16 last 162.47) - open/close 35098 (time 1734259222.67 total 204.16 last 170.72) - open/close 36914 (time 1734259232.67 total 214.16 last 181.58) - open/close 38822 (time 1734259242.67 total 224.16 last 190.72) - open/close 40000 (time 1734259248.10 total 229.59 last 217.24) - open/close 42346 (time 1734259258.10 total 239.59 last 234.58) - open/close 44377 (time 1734259268.10 total 249.59 last 203.00) - open/close 46090 (time 1734259278.10 total 259.60 last 171.25) - open/close 48056 (time 1734259288.11 total 269.60 last 196.55) - open/close 49772 (time 1734259298.11 total 279.60 last 171.54) - open/close 51581 (time 1734259308.11 total 289.60 last 180.89) - open/close 53316 (time 1734259318.11 total 299.60 last 173.47) - open/close 55356 (time 1734259328.11 total 309.61 last 203.95) - open/close 57398 (time 1734259338.12 total 319.61 last 204.16) - open/close 59417 (time 1734259348.12 total 329.61 last 201.88) - open/close 60000 (time 1734259351.59 total 333.08 last 167.76) - open/close 61721 (time 1734259361.60 total 343.09 last 172.05) - open/close 63500 (time 1734259371.60 total 353.09 last 177.81) total: 64768 open/close in 360.02 seconds: 179.90 ops/second - /unlink 3111 (time 1734259389.97 total 10.00 last 311.00) - /unlink 6384 (time 1734259399.97 total 20.00 last 327.26) - /unlink 9463 (time 1734259409.97 total 30.01 last 307.86) - /unlink 12519 (time 1734259419.97 total 40.01 last 305.56) - /unlink 15713 (time 1734259429.97 total 50.01 last 319.39) - /unlink 18795 (time 1734259439.97 total 60.01 last 308.14) - /unlink 20000 (time 1734259443.74 total 63.78 last 319.61) - /unlink 23206 (time 1734259453.74 total 73.78 last 320.52) - /unlink 26226 (time 1734259463.75 total 83.79 last 301.81) - /unlink 29406 (time 1734259473.75 total 93.79 last 317.99) - /unlink 32874 (time 1734259483.75 total 103.79 last 346.69) - /unlink 36378 (time 1734259493.76 total 113.80 last 350.28) - /unlink 39613 (time 1734259503.76 total 123.80 last 323.44) - /unlink 42698 (time 1734259513.76 total 133.80 last 308.39) - /unlink 46323 (time 1734259523.76 total 143.80 last 362.45) - /unlink 50000 (time 1734259532.13 total 152.17 last 439.48) - /unlink 53758 (time 1734259542.13 total 162.17 last 375.72) - /unlink 57172 (time 1734259552.13 total 172.17 last 341.39) - /unlink 60000 (time 1734259559.75 total 179.79 last 371.30) - /unlink 63510 (time 1734259569.75 total 189.79 last 350.96) total: 64768 /unlink in 193.74 seconds: 334.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2358 (time 1734259602.96 total 10.00 last 235.71) - open/close 4911 (time 1734259612.96 total 20.00 last 255.28) - open/close 7477 (time 1734259622.96 total 30.01 last 256.53) - open/close 9805 (time 1734259632.97 total 40.01 last 232.73) - open/close 12164 (time 1734259642.97 total 50.01 last 235.80) - open/close 14416 (time 1734259652.97 total 60.02 last 225.18) - open/close 16644 (time 1734259662.98 total 70.02 last 222.71) - open/close 18898 (time 1734259672.98 total 80.02 last 225.32) - open/close 20000 (time 1734259677.70 total 84.75 last 233.21) - open/close 22351 (time 1734259687.71 total 94.75 last 235.02) - open/close 24697 (time 1734259697.80 total 104.85 last 232.39) - open/close 27336 (time 1734259707.80 total 114.85 last 263.85) - open/close 29772 (time 1734259717.80 total 124.85 last 243.58) - open/close 31992 (time 1734259727.81 total 134.85 last 221.98) - open/close 34053 (time 1734259737.81 total 144.85 last 206.06) - open/close 36306 (time 1734259747.81 total 154.86 last 225.21) - open/close 38332 (time 1734259757.81 total 164.86 last 202.60) - open/close 40000 (time 1734259765.46 total 172.51 last 217.99) - open/close 42327 (time 1734259775.47 total 182.51 last 232.59) - open/close 44741 (time 1734259785.47 total 192.51 last 241.38) - open/close 47150 (time 1734259795.47 total 202.52 last 240.84) - open/close 49163 (time 1734259805.48 total 212.52 last 201.19) - open/close 50000 (time 1734259809.76 total 216.80 last 195.62) - open/close 52126 (time 1734259819.76 total 226.80 last 212.55) - open/close 54644 (time 1734259829.76 total 236.80 last 251.75) - open/close 57167 (time 1734259839.76 total 246.81 last 252.24) - open/close 59862 (time 1734259849.77 total 256.81 last 269.45) - open/close 62451 (time 1734259859.77 total 266.81 last 258.81) total: 64768 open/close in 276.39 seconds: 234.33 ops/second - /unlink 4066 (time 1734259880.60 total 10.00 last 406.51) - /unlink 7821 (time 1734259890.60 total 20.00 last 375.47) - /unlink 10000 (time 1734259896.84 total 26.24 last 349.24) - /unlink 13751 (time 1734259906.84 total 36.24 last 375.07) - /unlink 18011 (time 1734259916.84 total 46.24 last 425.98) - /unlink 20000 (time 1734259921.10 total 50.51 last 466.50) - /unlink 24339 (time 1734259931.10 total 60.51 last 433.89) - /unlink 28627 (time 1734259941.10 total 70.51 last 428.73) - /unlink 30000 (time 1734259944.51 total 73.92 last 402.87) - /unlink 34088 (time 1734259954.51 total 83.92 last 408.78) - /unlink 38040 (time 1734259964.51 total 93.92 last 395.19) - /unlink 40000 (time 1734259969.45 total 98.86 last 396.91) - /unlink 44178 (time 1734259979.45 total 108.86 last 417.80) - /unlink 48561 (time 1734259989.45 total 118.86 last 438.18) - /unlink 50000 (time 1734259992.78 total 122.19 last 431.97) - /unlink 55313 (time 1734260002.78 total 132.19 last 531.23) - /unlink 60000 (time 1734260010.92 total 140.33 last 575.93) total: 64768 /unlink in 149.45 seconds: 433.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2622 (time 1734260048.58 total 10.00 last 262.13) - open/close 5917 (time 1734260058.58 total 20.00 last 329.47) - open/close 9444 (time 1734260068.58 total 30.00 last 352.68) - open/close 12448 (time 1734260078.59 total 40.01 last 300.31) - open/close 15418 (time 1734260088.59 total 50.01 last 296.90) - open/close 18365 (time 1734260098.59 total 60.01 last 294.60) - open/close 20000 (time 1734260104.37 total 65.79 last 283.03) - open/close 23084 (time 1734260114.37 total 75.80 last 308.26) - open/close 25855 (time 1734260124.37 total 85.80 last 277.09) - open/close 28569 (time 1734260134.38 total 95.80 last 271.37) - open/close 30000 (time 1734260139.90 total 101.32 last 258.98) - open/close 33044 (time 1734260149.90 total 111.32 last 304.35) - open/close 36212 (time 1734260159.91 total 121.33 last 316.73) - open/close 39466 (time 1734260169.91 total 131.33 last 325.35) - open/close 42724 (time 1734260179.91 total 141.33 last 325.72) - open/close 45896 (time 1734260189.91 total 151.33 last 317.14) - open/close 48870 (time 1734260199.91 total 161.33 last 297.33) - open/close 50000 (time 1734260203.70 total 165.12 last 298.54) - open/close 52961 (time 1734260213.70 total 175.12 last 296.06) - open/close 55863 (time 1734260223.70 total 185.12 last 290.09) - open/close 58626 (time 1734260233.72 total 195.14 last 275.87) - open/close 60000 (time 1734260238.78 total 200.20 last 271.58) - open/close 62802 (time 1734260248.78 total 210.20 last 280.19) total: 64768 open/close in 216.50 seconds: 299.16 ops/second - /unlink 5873 (time 1734260266.02 total 10.00 last 587.26) - /unlink 10000 (time 1734260273.85 total 17.83 last 527.15) - /unlink 14635 (time 1734260283.85 total 27.83 last 463.50) - /unlink 19569 (time 1734260293.85 total 37.83 last 493.38) - /unlink 23798 (time 1734260303.86 total 47.83 last 422.81) - /unlink 28546 (time 1734260313.86 total 57.83 last 474.72) - /unlink 30000 (time 1734260317.15 total 61.13 last 441.24) - /unlink 34936 (time 1734260327.15 total 71.13 last 493.60) - /unlink 40000 (time 1734260337.04 total 81.01 last 512.40) - /unlink 45125 (time 1734260347.04 total 91.01 last 512.47) - /unlink 50000 (time 1734260355.94 total 99.92 last 547.41) - /unlink 54821 (time 1734260365.94 total 109.92 last 482.02) - /unlink 59819 (time 1734260375.94 total 119.92 last 499.76) - /unlink 64538 (time 1734260385.94 total 129.92 last 471.89) total: 64768 /unlink in 130.41 seconds: 496.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg324-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg324-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg324-server unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2