== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:11:55 (1734430315) Stopping clients: oleg341-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg341-client.virtnet /mnt/lustre2 (opts:-f) oleg341-server: oleg341-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg341-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg341-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1423 (time 1734430366.66 total 10.00 last 142.23) - open/close 2801 (time 1734430376.66 total 20.01 last 137.79) - open/close 4039 (time 1734430386.66 total 30.01 last 123.80) - open/close 5516 (time 1734430396.66 total 40.01 last 147.69) - open/close 7156 (time 1734430406.67 total 50.01 last 163.89) - open/close 8873 (time 1734430416.67 total 60.01 last 171.69) - open/close 10000 (time 1734430423.72 total 67.06 last 159.93) - open/close 11602 (time 1734430433.72 total 77.07 last 160.12) - open/close 13028 (time 1734430443.73 total 87.07 last 142.54) - open/close 14265 (time 1734430453.73 total 97.07 last 123.67) - open/close 15732 (time 1734430463.73 total 107.07 last 146.68) - open/close 17065 (time 1734430473.73 total 117.07 last 133.29) - open/close 18403 (time 1734430483.73 total 127.08 last 133.77) - open/close 19712 (time 1734430493.74 total 137.08 last 130.81) - open/close 20000 (time 1734430495.94 total 139.28 last 130.99) - open/close 21458 (time 1734430505.94 total 149.28 last 145.77) - open/close 22959 (time 1734430515.95 total 159.29 last 149.98) - open/close 24273 (time 1734430525.95 total 169.29 last 131.38) - open/close 25699 (time 1734430535.95 total 179.29 last 142.57) - open/close 27111 (time 1734430545.96 total 189.30 last 141.14) - open/close 28386 (time 1734430555.96 total 199.30 last 127.47) - open/close 29695 (time 1734430565.96 total 209.31 last 130.84) - open/close 30000 (time 1734430568.33 total 211.67 last 128.96) - open/close 31330 (time 1734430578.34 total 221.68 last 132.88) - open/close 32556 (time 1734430588.34 total 231.68 last 122.58) - open/close 33812 (time 1734430598.35 total 241.69 last 125.47) - open/close 35150 (time 1734430608.35 total 251.69 last 133.79) - open/close 36429 (time 1734430618.36 total 261.70 last 127.82) - open/close 37738 (time 1734430628.36 total 271.70 last 130.85) - open/close 38975 (time 1734430638.37 total 281.71 last 123.61) - open/close 40000 (time 1734430646.71 total 290.06 last 122.82) - open/close 41262 (time 1734430656.72 total 300.06 last 126.14) - open/close 42574 (time 1734430666.72 total 310.06 last 131.18) - open/close 43980 (time 1734430676.72 total 320.07 last 140.55) - open/close 45325 (time 1734430686.72 total 330.07 last 134.47) - open/close 46684 (time 1734430696.73 total 340.08 last 135.79) - open/close 48063 (time 1734430706.73 total 350.08 last 137.86) - open/close 49313 (time 1734430716.74 total 360.08 last 124.94) - open/close 50000 (time 1734430721.76 total 365.11 last 136.73) - open/close 51427 (time 1734430731.77 total 375.11 last 142.67) - open/close 52975 (time 1734430741.77 total 385.12 last 154.71) - open/close 54290 (time 1734430751.78 total 395.12 last 131.45) - open/close 55769 (time 1734430761.78 total 405.13 last 147.80) - open/close 57253 (time 1734430771.78 total 415.13 last 148.36) - open/close 58626 (time 1734430781.79 total 425.13 last 137.28) - open/close 60000 (time 1734430791.31 total 434.65 last 144.30) - open/close 61323 (time 1734430801.31 total 444.65 last 132.29) - open/close 62755 (time 1734430811.31 total 454.65 last 143.19) - open/close 64113 (time 1734430821.31 total 464.66 last 135.78) total: 64768 open/close in 469.14 seconds: 138.06 ops/second - /unlink 3110 (time 1734430837.03 total 10.00 last 310.88) - /unlink 6145 (time 1734430847.03 total 20.00 last 303.48) - /unlink 9493 (time 1734430857.03 total 30.01 last 334.73) - /unlink 12712 (time 1734430867.03 total 40.01 last 321.83) - /unlink 15822 (time 1734430877.03 total 50.01 last 310.97) - /unlink 18697 (time 1734430887.04 total 60.01 last 287.40) - /unlink 20000 (time 1734430891.50 total 64.47 last 292.24) - /unlink 22915 (time 1734430901.50 total 74.47 last 291.50) - /unlink 26553 (time 1734430911.50 total 84.47 last 363.74) - /unlink 29329 (time 1734430921.50 total 94.47 last 277.57) - /unlink 30000 (time 1734430924.53 total 97.51 last 221.13) - /unlink 32316 (time 1734430934.54 total 107.52 last 231.40) - /unlink 34659 (time 1734430944.55 total 117.52 last 234.23) - /unlink 37560 (time 1734430954.55 total 127.52 last 290.04) - /unlink 40000 (time 1734430963.09 total 136.06 last 285.66) - /unlink 42473 (time 1734430973.09 total 146.06 last 247.30) - /unlink 45189 (time 1734430983.09 total 156.07 last 271.55) - /unlink 48571 (time 1734430993.09 total 166.07 last 338.12) - /unlink 50000 (time 1734430997.14 total 170.11 last 353.17) - /unlink 53307 (time 1734431007.14 total 180.12 last 330.65) - /unlink 56755 (time 1734431017.15 total 190.12 last 344.62) - /unlink 59960 (time 1734431027.15 total 200.12 last 320.41) - /unlink 63261 (time 1734431037.15 total 210.13 last 330.02) total: 64768 /unlink in 214.85 seconds: 301.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1925 (time 1734431067.01 total 10.00 last 192.46) - open/close 3411 (time 1734431077.02 total 20.01 last 148.54) - open/close 4899 (time 1734431087.03 total 30.01 last 148.70) - open/close 6478 (time 1734431097.03 total 40.02 last 157.86) - open/close 8221 (time 1734431107.03 total 50.02 last 174.23) - open/close 9795 (time 1734431117.04 total 60.02 last 157.33) - open/close 11555 (time 1734431127.04 total 70.03 last 175.93) - open/close 13474 (time 1734431137.04 total 80.03 last 191.85) - open/close 15538 (time 1734431147.05 total 90.03 last 206.35) - open/close 17777 (time 1734431157.05 total 100.04 last 223.84) - open/close 20000 (time 1734431165.56 total 108.55 last 261.18) - open/close 22142 (time 1734431175.56 total 118.55 last 214.16) - open/close 24085 (time 1734431185.56 total 128.55 last 194.27) - open/close 26457 (time 1734431195.57 total 138.56 last 237.09) - open/close 28234 (time 1734431205.57 total 148.56 last 177.65) - open/close 30000 (time 1734431215.52 total 158.51 last 177.45) - open/close 31684 (time 1734431225.53 total 168.51 last 168.33) - open/close 33151 (time 1734431235.53 total 178.52 last 146.61) - open/close 34518 (time 1734431245.53 total 188.52 last 136.68) - open/close 35900 (time 1734431255.54 total 198.52 last 138.16) - open/close 37483 (time 1734431265.54 total 208.53 last 158.27) - open/close 39130 (time 1734431275.54 total 218.53 last 164.70) - open/close 40000 (time 1734431281.53 total 224.52 last 145.22) - open/close 41507 (time 1734431291.53 total 234.52 last 150.70) - open/close 43403 (time 1734431301.53 total 244.52 last 189.57) - open/close 45115 (time 1734431311.54 total 254.52 last 171.13) - open/close 46955 (time 1734431321.54 total 264.53 last 183.94) - open/close 48882 (time 1734431331.55 total 274.53 last 192.57) - open/close 50000 (time 1734431337.35 total 280.33 last 192.74) - open/close 52159 (time 1734431347.35 total 290.34 last 215.87) - open/close 54044 (time 1734431357.35 total 300.34 last 188.47) - open/close 55996 (time 1734431367.35 total 310.34 last 195.17) - open/close 57614 (time 1734431377.36 total 320.35 last 161.65) - open/close 59158 (time 1734431387.36 total 330.35 last 154.37) - open/close 60000 (time 1734431392.93 total 335.91 last 151.33) - open/close 61617 (time 1734431402.93 total 345.92 last 161.65) - open/close 63078 (time 1734431412.93 total 355.92 last 146.07) - open/close 64393 (time 1734431422.94 total 365.93 last 131.39) total: 64768 open/close in 368.75 seconds: 175.64 ops/second - /unlink 2509 (time 1734431437.52 total 10.00 last 250.85) - /unlink 5884 (time 1734431447.52 total 20.00 last 337.48) - /unlink 9223 (time 1734431457.52 total 30.00 last 333.84) - /unlink 10000 (time 1734431460.25 total 32.73 last 284.82) - /unlink 12680 (time 1734431470.25 total 42.73 last 267.98) - /unlink 14879 (time 1734431480.25 total 52.74 last 219.84) - /unlink 17337 (time 1734431490.26 total 62.74 last 245.79) - /unlink 20000 (time 1734431498.89 total 71.38 last 308.26) - /unlink 23257 (time 1734431508.89 total 81.38 last 325.69) - /unlink 27107 (time 1734431518.90 total 91.38 last 384.95) - /unlink 30000 (time 1734431527.72 total 100.20 last 327.93) - /unlink 33907 (time 1734431537.72 total 110.20 last 390.66) - /unlink 36968 (time 1734431547.72 total 120.20 last 305.98) - /unlink 39723 (time 1734431557.72 total 130.20 last 275.49) - /unlink 43084 (time 1734431567.72 total 140.20 last 336.08) - /unlink 46543 (time 1734431577.72 total 150.21 last 345.87) - /unlink 49874 (time 1734431587.73 total 160.21 last 333.07) - /unlink 53551 (time 1734431597.73 total 170.21 last 367.69) - /unlink 58022 (time 1734431607.73 total 180.21 last 447.08) - /unlink 60000 (time 1734431611.16 total 183.64 last 575.66) total: 64768 /unlink in 191.57 seconds: 338.09 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2297 (time 1734431647.94 total 10.01 last 229.56) - open/close 4284 (time 1734431657.94 total 20.01 last 198.70) - open/close 6040 (time 1734431667.94 total 30.01 last 175.55) - open/close 7959 (time 1734431677.95 total 40.02 last 191.71) - open/close 9687 (time 1734431687.96 total 50.03 last 172.69) - open/close 10000 (time 1734431690.26 total 52.33 last 135.82) - open/close 11302 (time 1734431700.27 total 62.34 last 130.08) - open/close 12777 (time 1734431710.28 total 72.35 last 147.41) - open/close 14156 (time 1734431720.28 total 82.35 last 137.83) - open/close 15515 (time 1734431730.28 total 92.35 last 135.88) - open/close 17815 (time 1734431740.29 total 102.35 last 229.96) - open/close 20000 (time 1734431749.89 total 111.95 last 227.60) - open/close 22459 (time 1734431759.89 total 121.96 last 245.81) - open/close 24663 (time 1734431769.89 total 131.96 last 220.34) - open/close 26484 (time 1734431779.89 total 141.96 last 182.07) - open/close 28710 (time 1734431789.90 total 151.97 last 222.52) - open/close 30000 (time 1734431795.34 total 157.41 last 237.11) - open/close 31648 (time 1734431805.34 total 167.41 last 164.78) - open/close 33268 (time 1734431815.34 total 177.41 last 161.93) - open/close 35702 (time 1734431825.35 total 187.41 last 243.37) - open/close 37567 (time 1734431835.35 total 197.42 last 186.40) - open/close 39282 (time 1734431845.36 total 207.43 last 171.39) - open/close 40000 (time 1734431850.22 total 212.29 last 147.66) - open/close 41809 (time 1734431860.22 total 222.29 last 180.87) - open/close 43959 (time 1734431870.23 total 232.29 last 214.91) - open/close 46090 (time 1734431880.23 total 242.29 last 213.08) - open/close 47901 (time 1734431890.23 total 252.29 last 181.09) - open/close 50000 (time 1734431899.71 total 261.78 last 221.34) - open/close 52183 (time 1734431909.71 total 271.78 last 218.26) - open/close 54266 (time 1734431919.71 total 281.78 last 208.29) - open/close 56239 (time 1734431929.71 total 291.78 last 197.26) - open/close 58475 (time 1734431939.72 total 301.79 last 223.51) - open/close 60000 (time 1734431947.18 total 309.25 last 204.30) - open/close 62236 (time 1734431957.19 total 319.25 last 223.54) - open/close 63838 (time 1734431967.19 total 329.26 last 160.18) total: 64768 open/close in 334.77 seconds: 193.47 ops/second - /unlink 2952 (time 1734431983.92 total 10.00 last 295.13) - /unlink 5638 (time 1734431993.92 total 20.00 last 268.58) - /unlink 8669 (time 1734432003.92 total 30.00 last 303.05) - /unlink 10000 (time 1734432009.24 total 35.32 last 250.26) - /unlink 12451 (time 1734432019.24 total 45.33 last 244.98) - /unlink 15861 (time 1734432029.24 total 55.33 last 340.99) - /unlink 18950 (time 1734432039.24 total 65.33 last 308.84) - /unlink 20000 (time 1734432042.75 total 68.84 last 299.34) - /unlink 22871 (time 1734432052.75 total 78.84 last 287.09) - /unlink 26393 (time 1734432062.75 total 88.84 last 352.14) - /unlink 29227 (time 1734432072.75 total 98.84 last 283.39) - /unlink 30000 (time 1734432075.05 total 101.13 last 336.96) - /unlink 33632 (time 1734432085.05 total 111.13 last 363.20) - /unlink 36707 (time 1734432095.05 total 121.14 last 307.42) - /unlink 39571 (time 1734432105.05 total 131.14 last 286.36) - /unlink 42402 (time 1734432115.06 total 141.14 last 282.99) - /unlink 45188 (time 1734432125.06 total 151.15 last 278.48) - /unlink 47811 (time 1734432135.07 total 161.15 last 262.19) - /unlink 50000 (time 1734432141.91 total 168.00 last 319.70) - /unlink 52954 (time 1734432151.91 total 178.00 last 295.37) - /unlink 56330 (time 1734432161.91 total 188.00 last 337.59) - /unlink 60000 (time 1734432169.93 total 196.02 last 457.55) - /unlink 64350 (time 1734432179.93 total 206.02 last 434.99) total: 64768 /unlink in 206.76 seconds: 313.26 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2464 (time 1734432206.29 total 10.00 last 246.37) - open/close 5489 (time 1734432216.30 total 20.00 last 302.41) - open/close 8304 (time 1734432226.30 total 30.01 last 281.48) - open/close 10000 (time 1734432233.45 total 37.16 last 237.16) - open/close 12765 (time 1734432243.45 total 47.16 last 276.50) - open/close 15545 (time 1734432253.45 total 57.16 last 277.98) - open/close 18725 (time 1734432263.45 total 67.16 last 317.99) - open/close 20000 (time 1734432267.84 total 71.55 last 290.52) - open/close 22907 (time 1734432277.84 total 81.55 last 290.68) - open/close 25720 (time 1734432287.87 total 91.58 last 280.50) - open/close 28135 (time 1734432297.87 total 101.58 last 241.35) - open/close 30000 (time 1734432306.05 total 109.76 last 228.10) - open/close 32763 (time 1734432316.05 total 119.76 last 276.23) - open/close 34978 (time 1734432326.06 total 129.76 last 221.42) - open/close 37501 (time 1734432336.06 total 139.76 last 252.29) - open/close 40000 (time 1734432344.66 total 148.37 last 290.34) - open/close 42470 (time 1734432354.67 total 158.38 last 246.89) - open/close 45016 (time 1734432364.67 total 168.38 last 254.45) - open/close 47924 (time 1734432374.68 total 178.38 last 290.76) - open/close 49801 (time 1734432384.68 total 188.38 last 187.69) - open/close 52241 (time 1734432394.68 total 198.38 last 243.98) - open/close 54141 (time 1734432404.68 total 208.39 last 189.89) - open/close 56555 (time 1734432414.69 total 218.39 last 241.31) - open/close 59058 (time 1734432424.69 total 228.39 last 250.29) - open/close 60000 (time 1734432428.33 total 232.03 last 258.91) - open/close 62581 (time 1734432438.33 total 242.03 last 258.08) - open/close 64682 (time 1734432448.33 total 252.04 last 209.99) total: 64768 open/close in 252.35 seconds: 256.66 ops/second - /unlink 4163 (time 1734432459.73 total 10.00 last 416.21) - /unlink 8335 (time 1734432469.73 total 20.00 last 417.18) - /unlink 10000 (time 1734432473.32 total 23.59 last 464.60) - /unlink 13872 (time 1734432483.32 total 33.59 last 387.18) - /unlink 17788 (time 1734432493.32 total 43.59 last 391.57) - /unlink 20000 (time 1734432499.88 total 50.15 last 337.17) - /unlink 24818 (time 1734432509.88 total 60.15 last 481.73) - /unlink 29574 (time 1734432519.88 total 70.15 last 475.57) - /unlink 34161 (time 1734432529.88 total 80.15 last 458.59) - /unlink 37877 (time 1734432539.88 total 90.15 last 371.55) - /unlink 40000 (time 1734432545.19 total 95.46 last 399.77) - /unlink 44913 (time 1734432555.20 total 105.47 last 491.26) - /unlink 50000 (time 1734432563.81 total 114.08 last 590.70) - /unlink 55737 (time 1734432573.81 total 124.08 last 573.64) - /unlink 60000 (time 1734432579.99 total 130.26 last 689.51) total: 64768 /unlink in 138.15 seconds: 468.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3391 (time 1734432617.46 total 10.00 last 339.03) - open/close 7200 (time 1734432627.46 total 20.00 last 380.89) - open/close 10000 (time 1734432634.67 total 27.21 last 388.37) - open/close 14635 (time 1734432644.67 total 37.21 last 463.48) - open/close 19173 (time 1734432654.67 total 47.21 last 453.74) - open/close 20000 (time 1734432656.98 total 49.52 last 358.16) - open/close 23916 (time 1734432666.98 total 59.52 last 391.58) - open/close 27643 (time 1734432676.98 total 69.52 last 372.65) - open/close 30000 (time 1734432682.81 total 75.35 last 404.35) - open/close 32955 (time 1734432692.82 total 85.36 last 295.39) - open/close 35999 (time 1734432702.82 total 95.36 last 304.31) - open/close 38621 (time 1734432712.82 total 105.36 last 262.17) - open/close 40000 (time 1734432717.03 total 109.57 last 327.91) - open/close 44757 (time 1734432727.03 total 119.57 last 475.69) - open/close 49575 (time 1734432737.03 total 129.57 last 481.73) - open/close 53985 (time 1734432747.03 total 139.57 last 440.94) - open/close 58402 (time 1734432757.03 total 149.57 last 441.65) - open/close 60000 (time 1734432760.68 total 153.22 last 437.68) - open/close 64563 (time 1734432770.68 total 163.22 last 456.23) total: 64768 open/close in 163.68 seconds: 395.70 ops/second - /unlink 7447 (time 1734432782.03 total 10.00 last 744.65) - /unlink 10000 (time 1734432784.79 total 12.76 last 924.06) - /unlink 19515 (time 1734432794.80 total 22.76 last 951.39) - /unlink 28590 (time 1734432804.80 total 32.77 last 907.45) - /unlink 36850 (time 1734432814.80 total 42.77 last 825.96) - /unlink 40000 (time 1734432818.55 total 46.52 last 838.70) - /unlink 47735 (time 1734432828.55 total 56.52 last 773.49) - /unlink 50000 (time 1734432831.74 total 59.71 last 711.49) - /unlink 57627 (time 1734432841.74 total 69.71 last 762.67) - /unlink 60000 (time 1734432844.80 total 72.76 last 775.74) total: 64768 /unlink in 79.37 seconds: 816.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server unloading modules via unload_modules_local on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2