== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:16:21 (1734257781) Stopping clients: oleg358-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg358-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 oleg358-server: oleg358-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: detected 4 online CPUs by sysfs oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg358-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg358-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg358-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock 192.168.203.158@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16200 392 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2191 (time 1734257843.39 total 10.00 last 219.10) - open/close 4731 (time 1734257853.39 total 20.00 last 253.98) - open/close 7095 (time 1734257863.39 total 30.00 last 236.32) - open/close 8919 (time 1734257873.39 total 40.00 last 182.40) - open/close 10000 (time 1734257879.58 total 46.19 last 174.75) - open/close 11848 (time 1734257889.58 total 56.19 last 184.79) - open/close 13534 (time 1734257899.59 total 66.20 last 168.51) - open/close 15487 (time 1734257909.59 total 76.20 last 195.16) - open/close 17112 (time 1734257919.60 total 86.21 last 162.46) - open/close 18879 (time 1734257929.60 total 96.21 last 176.66) - open/close 20000 (time 1734257936.97 total 103.58 last 152.07) - open/close 21581 (time 1734257946.97 total 113.58 last 158.05) - open/close 23275 (time 1734257956.98 total 123.59 last 169.34) - open/close 25127 (time 1734257966.98 total 133.59 last 185.14) - open/close 26860 (time 1734257976.98 total 143.59 last 173.24) - open/close 28731 (time 1734257986.99 total 153.60 last 186.97) - open/close 30000 (time 1734257994.29 total 160.90 last 173.78) - open/close 31745 (time 1734258004.29 total 170.90 last 174.45) - open/close 33582 (time 1734258014.30 total 180.90 last 183.70) - open/close 35254 (time 1734258024.30 total 190.91 last 167.13) - open/close 36766 (time 1734258034.30 total 200.91 last 151.18) - open/close 38243 (time 1734258044.30 total 210.91 last 147.65) - open/close 39841 (time 1734258054.31 total 220.92 last 159.76) - open/close 41452 (time 1734258064.33 total 230.93 last 160.81) - open/close 43025 (time 1734258074.33 total 240.94 last 157.22) - open/close 44522 (time 1734258084.33 total 250.94 last 149.66) - open/close 45984 (time 1734258094.34 total 260.95 last 146.14) - open/close 47339 (time 1734258104.34 total 270.95 last 135.49) - open/close 48615 (time 1734258114.34 total 280.95 last 127.55) - open/close 49991 (time 1734258124.35 total 290.96 last 137.53) - open/close 51400 (time 1734258134.35 total 300.96 last 140.86) - open/close 52873 (time 1734258144.35 total 310.96 last 147.26) - open/close 54387 (time 1734258154.36 total 320.97 last 151.34) - open/close 55675 (time 1734258164.37 total 330.98 last 128.69) - open/close 57125 (time 1734258174.37 total 340.98 last 144.93) - open/close 58421 (time 1734258184.37 total 350.98 last 129.59) - open/close 59807 (time 1734258194.37 total 360.98 last 138.55) - open/close 61267 (time 1734258204.38 total 370.99 last 145.94) - open/close 62669 (time 1734258214.38 total 380.99 last 140.12) - open/close 64293 (time 1734258224.39 total 391.00 last 162.34) total: 64768 open/close in 394.08 seconds: 164.35 ops/second - /unlink 2627 (time 1734258239.10 total 10.00 last 262.66) - /unlink 5263 (time 1734258249.10 total 20.01 last 263.50) - /unlink 7629 (time 1734258259.10 total 30.01 last 236.53) - /unlink 10000 (time 1734258269.10 total 40.01 last 237.17) - /unlink 12374 (time 1734258279.11 total 50.01 last 237.29) - /unlink 15047 (time 1734258289.11 total 60.01 last 267.25) - /unlink 17952 (time 1734258299.11 total 70.01 last 290.47) - /unlink 20000 (time 1734258305.98 total 76.88 last 298.05) - /unlink 23644 (time 1734258315.98 total 86.89 last 364.32) - /unlink 27023 (time 1734258325.98 total 96.89 last 337.83) - /unlink 30000 (time 1734258334.76 total 105.67 last 339.09) - /unlink 32879 (time 1734258344.76 total 115.67 last 287.87) - /unlink 35815 (time 1734258354.77 total 125.67 last 293.53) - /unlink 38722 (time 1734258364.77 total 135.67 last 290.67) - /unlink 40000 (time 1734258369.19 total 140.10 last 288.79) - /unlink 42730 (time 1734258379.19 total 150.10 last 272.99) - /unlink 45428 (time 1734258389.20 total 160.10 last 269.68) - /unlink 48267 (time 1734258399.20 total 170.10 last 283.88) - /unlink 50000 (time 1734258405.30 total 176.21 last 283.83) - /unlink 53163 (time 1734258415.31 total 186.21 last 316.18) - /unlink 56313 (time 1734258425.31 total 196.21 last 315.00) - /unlink 59111 (time 1734258435.31 total 206.22 last 279.75) - /unlink 60000 (time 1734258438.61 total 209.52 last 269.22) - /unlink 62762 (time 1734258448.61 total 219.52 last 276.14) total: 64768 /unlink in 226.64 seconds: 285.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1768 (time 1734258483.37 total 10.00 last 176.76) - open/close 3484 (time 1734258493.37 total 20.01 last 171.47) - open/close 5643 (time 1734258503.38 total 30.01 last 215.86) - open/close 7780 (time 1734258513.38 total 40.01 last 213.67) - open/close 9661 (time 1734258523.38 total 50.02 last 188.03) - open/close 11604 (time 1734258533.38 total 60.02 last 194.30) - open/close 13841 (time 1734258543.38 total 70.02 last 223.61) - open/close 16022 (time 1734258553.39 total 80.02 last 218.02) - open/close 17900 (time 1734258563.39 total 90.03 last 187.69) - open/close 20000 (time 1734258572.39 total 99.03 last 233.46) - open/close 21934 (time 1734258582.39 total 109.03 last 193.32) - open/close 23643 (time 1734258592.39 total 119.03 last 170.88) - open/close 25340 (time 1734258602.40 total 129.03 last 169.64) - open/close 27112 (time 1734258612.40 total 139.03 last 177.19) - open/close 28908 (time 1734258622.40 total 149.04 last 179.54) - open/close 30000 (time 1734258627.82 total 154.46 last 201.49) - open/close 31824 (time 1734258637.83 total 164.46 last 182.27) - open/close 33377 (time 1734258647.84 total 174.47 last 155.17) - open/close 35004 (time 1734258657.84 total 184.48 last 162.66) - open/close 36716 (time 1734258667.84 total 194.48 last 171.17) - open/close 38247 (time 1734258677.84 total 204.48 last 153.10) - open/close 39839 (time 1734258687.84 total 214.48 last 159.18) - open/close 41472 (time 1734258697.84 total 224.48 last 163.27) - open/close 43005 (time 1734258707.85 total 234.48 last 153.27) - open/close 44558 (time 1734258717.85 total 244.49 last 155.24) - open/close 46285 (time 1734258727.86 total 254.49 last 172.62) - open/close 48005 (time 1734258737.86 total 264.50 last 171.90) - open/close 49804 (time 1734258747.86 total 274.50 last 179.87) - open/close 51559 (time 1734258757.86 total 284.50 last 175.47) - open/close 53321 (time 1734258767.87 total 294.50 last 176.18) - open/close 55182 (time 1734258777.87 total 304.50 last 186.10) - open/close 56765 (time 1734258787.87 total 314.51 last 158.24) - open/close 58408 (time 1734258797.87 total 324.51 last 164.27) - open/close 60000 (time 1734258807.63 total 334.26 last 163.20) - open/close 61816 (time 1734258817.63 total 344.27 last 181.55) - open/close 63594 (time 1734258827.64 total 354.27 last 177.69) total: 64768 open/close in 360.05 seconds: 179.89 ops/second - /unlink 3313 (time 1734258844.61 total 10.00 last 331.27) - /unlink 6660 (time 1734258854.62 total 20.01 last 334.54) - /unlink 10000 (time 1734258864.33 total 29.72 last 343.95) - /unlink 13322 (time 1734258874.33 total 39.72 last 332.11) - /unlink 16015 (time 1734258884.33 total 49.72 last 269.26) - /unlink 18766 (time 1734258894.33 total 59.72 last 275.09) - /unlink 20000 (time 1734258898.89 total 64.28 last 270.90) - /unlink 22634 (time 1734258908.89 total 74.28 last 263.39) - /unlink 25149 (time 1734258918.89 total 84.28 last 251.41) - /unlink 27720 (time 1734258928.89 total 94.28 last 257.06) - /unlink 30000 (time 1734258937.16 total 102.55 last 275.86) - /unlink 32773 (time 1734258947.16 total 112.55 last 277.27) - /unlink 35481 (time 1734258957.16 total 122.55 last 270.80) - /unlink 38475 (time 1734258967.16 total 132.55 last 299.29) - /unlink 40000 (time 1734258972.79 total 138.18 last 271.16) - /unlink 42894 (time 1734258982.79 total 148.18 last 289.30) - /unlink 46148 (time 1734258992.79 total 158.18 last 325.38) - /unlink 49432 (time 1734259002.79 total 168.18 last 328.34) - /unlink 53083 (time 1734259012.79 total 178.18 last 365.08) - /unlink 56306 (time 1734259022.79 total 188.18 last 322.28) - /unlink 59301 (time 1734259032.80 total 198.18 last 299.44) - /unlink 60000 (time 1734259034.99 total 200.37 last 319.23) - /unlink 62970 (time 1734259044.99 total 210.38 last 296.88) total: 64768 /unlink in 216.45 seconds: 299.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1592 (time 1734259080.23 total 10.00 last 159.18) - open/close 3245 (time 1734259090.23 total 20.00 last 165.27) - open/close 4869 (time 1734259100.24 total 30.00 last 162.37) - open/close 6389 (time 1734259110.24 total 40.01 last 151.95) - open/close 8295 (time 1734259120.24 total 50.01 last 190.56) - open/close 10000 (time 1734259129.80 total 59.57 last 178.41) - open/close 11667 (time 1734259139.80 total 69.57 last 166.62) - open/close 13309 (time 1734259149.81 total 79.57 last 164.16) - open/close 15062 (time 1734259159.81 total 89.58 last 175.26) - open/close 16900 (time 1734259169.81 total 99.58 last 183.75) - open/close 18806 (time 1734259179.81 total 109.58 last 190.58) - open/close 20000 (time 1734259186.49 total 116.26 last 178.69) - open/close 21612 (time 1734259196.50 total 126.26 last 161.17) - open/close 23377 (time 1734259206.50 total 136.26 last 176.49) - open/close 24968 (time 1734259216.50 total 146.27 last 159.05) - open/close 26668 (time 1734259226.50 total 156.27 last 169.99) - open/close 28449 (time 1734259236.51 total 166.27 last 178.00) - open/close 30000 (time 1734259244.22 total 173.99 last 201.01) - open/close 32426 (time 1734259254.22 total 183.99 last 242.54) - open/close 34597 (time 1734259264.23 total 193.99 last 217.07) - open/close 36367 (time 1734259274.23 total 204.00 last 176.91) - open/close 38250 (time 1734259284.23 total 214.00 last 188.28) - open/close 40000 (time 1734259292.86 total 222.63 last 202.89) - open/close 41744 (time 1734259302.86 total 232.63 last 174.36) - open/close 43527 (time 1734259312.86 total 242.63 last 178.25) - open/close 45290 (time 1734259322.87 total 252.63 last 176.24) - open/close 47495 (time 1734259332.87 total 262.63 last 220.50) - open/close 49598 (time 1734259342.87 total 272.64 last 210.27) - open/close 51394 (time 1734259352.87 total 282.64 last 179.53) - open/close 53132 (time 1734259362.88 total 292.65 last 173.66) - open/close 54903 (time 1734259372.88 total 302.65 last 177.03) - open/close 56689 (time 1734259382.88 total 312.65 last 178.58) - open/close 58466 (time 1734259392.89 total 322.66 last 177.64) - open/close 60000 (time 1734259401.14 total 330.90 last 185.98) - open/close 61786 (time 1734259411.14 total 340.91 last 178.55) - open/close 63525 (time 1734259421.14 total 350.91 last 173.90) total: 64768 open/close in 358.09 seconds: 180.87 ops/second - /unlink 3038 (time 1734259439.68 total 10.00 last 303.70) - /unlink 6179 (time 1734259449.68 total 20.00 last 314.09) - /unlink 9134 (time 1734259459.68 total 30.00 last 295.50) - /unlink 10000 (time 1734259462.28 total 32.60 last 333.30) - /unlink 13022 (time 1734259472.28 total 42.60 last 302.15) - /unlink 16338 (time 1734259482.28 total 52.60 last 331.56) - /unlink 19817 (time 1734259492.28 total 62.61 last 347.85) - /unlink 23080 (time 1734259502.28 total 72.61 last 326.30) - /unlink 26014 (time 1734259512.29 total 82.61 last 293.39) - /unlink 29615 (time 1734259522.29 total 92.61 last 360.08) - /unlink 33818 (time 1734259532.29 total 102.61 last 420.25) - /unlink 37479 (time 1734259542.29 total 112.61 last 365.99) - /unlink 40000 (time 1734259549.92 total 120.24 last 330.27) - /unlink 43638 (time 1734259559.92 total 130.25 last 363.75) - /unlink 47077 (time 1734259569.93 total 140.25 last 343.81) - /unlink 50000 (time 1734259578.97 total 149.29 last 323.23) - /unlink 53880 (time 1734259588.97 total 159.29 last 387.94) - /unlink 58003 (time 1734259598.97 total 169.29 last 412.24) - /unlink 60000 (time 1734259603.30 total 173.62 last 462.08) - /unlink 64725 (time 1734259613.30 total 183.62 last 472.45) total: 64768 /unlink in 183.70 seconds: 352.57 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2377 (time 1734259640.21 total 10.00 last 237.62) - open/close 4699 (time 1734259650.21 total 20.01 last 232.11) - open/close 6984 (time 1734259660.22 total 30.01 last 228.43) - open/close 9250 (time 1734259670.22 total 40.01 last 226.51) - open/close 10000 (time 1734259673.54 total 43.33 last 225.92) - open/close 12333 (time 1734259683.54 total 53.34 last 233.21) - open/close 14627 (time 1734259693.54 total 63.34 last 229.39) - open/close 17200 (time 1734259703.55 total 73.34 last 257.22) - open/close 19770 (time 1734259713.55 total 83.34 last 256.94) - open/close 22048 (time 1734259723.55 total 93.35 last 227.71) - open/close 24270 (time 1734259733.55 total 103.35 last 222.17) - open/close 26392 (time 1734259743.56 total 113.35 last 212.11) - open/close 28577 (time 1734259753.56 total 123.36 last 218.48) - open/close 30000 (time 1734259760.18 total 129.98 last 214.88) - open/close 32274 (time 1734259770.19 total 139.98 last 227.32) - open/close 34619 (time 1734259780.19 total 149.98 last 234.49) - open/close 37019 (time 1734259790.19 total 159.98 last 239.99) - open/close 39240 (time 1734259800.19 total 169.99 last 222.01) - open/close 40000 (time 1734259804.50 total 174.30 last 176.27) - open/close 42038 (time 1734259814.51 total 184.30 last 203.70) - open/close 44318 (time 1734259824.51 total 194.30 last 227.99) - open/close 46869 (time 1734259834.51 total 204.30 last 255.07) - open/close 49392 (time 1734259844.51 total 214.31 last 252.27) - open/close 50000 (time 1734259846.92 total 216.72 last 252.20) - open/close 52585 (time 1734259856.92 total 226.72 last 258.48) - open/close 55079 (time 1734259866.92 total 236.72 last 249.34) - open/close 57224 (time 1734259876.93 total 246.72 last 214.42) - open/close 59300 (time 1734259886.93 total 256.72 last 207.57) - open/close 60000 (time 1734259890.39 total 260.19 last 202.15) - open/close 62004 (time 1734259900.39 total 270.19 last 200.37) - open/close 64109 (time 1734259910.39 total 280.19 last 210.49) total: 64768 open/close in 283.07 seconds: 228.80 ops/second - /unlink 4400 (time 1734259924.33 total 10.00 last 439.95) - /unlink 8722 (time 1734259934.33 total 20.00 last 432.18) - /unlink 10000 (time 1734259937.26 total 22.93 last 436.65) - /unlink 13960 (time 1734259947.26 total 32.93 last 395.92) - /unlink 18082 (time 1734259957.26 total 42.93 last 412.09) - /unlink 20000 (time 1734259962.28 total 47.95 last 382.61) - /unlink 24017 (time 1734259972.28 total 57.95 last 401.65) - /unlink 28081 (time 1734259982.28 total 67.95 last 406.37) - /unlink 30000 (time 1734259986.40 total 72.07 last 465.75) - /unlink 34556 (time 1734259996.40 total 82.07 last 455.60) - /unlink 40000 (time 1734260006.13 total 91.80 last 559.21) - /unlink 45281 (time 1734260016.13 total 101.80 last 528.03) - /unlink 50000 (time 1734260025.54 total 111.21 last 501.73) - /unlink 55186 (time 1734260035.54 total 121.21 last 518.53) - /unlink 59925 (time 1734260045.54 total 131.21 last 473.81) total: 64768 /unlink in 140.15 seconds: 462.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3014 (time 1734260080.86 total 10.00 last 301.39) - open/close 6035 (time 1734260090.86 total 20.00 last 302.09) - open/close 8969 (time 1734260100.86 total 30.00 last 293.39) - open/close 10000 (time 1734260104.36 total 33.49 last 295.11) - open/close 13118 (time 1734260114.36 total 43.50 last 311.74) - open/close 15918 (time 1734260124.36 total 53.50 last 280.00) - open/close 18647 (time 1734260134.36 total 63.50 last 272.84) - open/close 20000 (time 1734260139.42 total 68.56 last 267.40) - open/close 23033 (time 1734260149.42 total 78.56 last 303.25) - open/close 26170 (time 1734260159.43 total 88.56 last 313.66) - open/close 29575 (time 1734260169.43 total 98.56 last 340.41) - open/close 32805 (time 1734260179.43 total 108.57 last 322.92) - open/close 36059 (time 1734260189.43 total 118.57 last 325.36) - open/close 39135 (time 1734260199.43 total 128.57 last 307.56) - open/close 40000 (time 1734260202.36 total 131.50 last 295.57) - open/close 43031 (time 1734260212.36 total 141.50 last 303.08) - open/close 45984 (time 1734260222.36 total 151.50 last 295.30) - open/close 48706 (time 1734260232.36 total 161.50 last 272.11) - open/close 50000 (time 1734260237.04 total 166.18 last 276.67) - open/close 52700 (time 1734260247.04 total 176.18 last 269.95) - open/close 55795 (time 1734260257.04 total 186.18 last 309.43) - open/close 58966 (time 1734260267.04 total 196.18 last 317.08) - open/close 60000 (time 1734260270.83 total 199.96 last 273.49) - open/close 62595 (time 1734260280.83 total 209.96 last 259.49) total: 64768 open/close in 217.99 seconds: 297.12 ops/second - /unlink 4336 (time 1734260299.84 total 10.00 last 433.48) - /unlink 8900 (time 1734260309.84 total 20.00 last 456.31) - /unlink 10000 (time 1734260311.98 total 22.14 last 514.27) - /unlink 14732 (time 1734260321.98 total 32.15 last 473.10) - /unlink 19964 (time 1734260331.98 total 42.15 last 523.15) - /unlink 25047 (time 1734260341.99 total 52.15 last 508.22) - /unlink 30000 (time 1734260351.59 total 61.75 last 515.67) - /unlink 35105 (time 1734260361.59 total 71.75 last 510.48) - /unlink 39958 (time 1734260371.59 total 81.75 last 485.24) - /unlink 44547 (time 1734260381.59 total 91.76 last 458.88) - /unlink 49102 (time 1734260391.60 total 101.76 last 455.42) - /unlink 50000 (time 1734260393.68 total 103.85 last 430.06) - /unlink 54283 (time 1734260403.68 total 113.85 last 428.26) - /unlink 58512 (time 1734260413.68 total 123.85 last 422.87) - /unlink 60000 (time 1734260416.76 total 126.92 last 484.37) total: 64768 /unlink in 136.72 seconds: 473.72 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server unloading modules via unload_modules_local on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2