== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:11:01 (1746144661) Stopping clients: oleg221-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg221-client.virtnet /mnt/lustre2 (opts:-f) oleg221-server: oleg221-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg221-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg221-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1210 (time 1746144715.67 total 10.00 last 120.99) - open/close 2664 (time 1746144725.67 total 20.00 last 145.40) - open/close 4241 (time 1746144735.68 total 30.00 last 157.65) - open/close 5962 (time 1746144745.68 total 40.01 last 172.05) - open/close 7415 (time 1746144755.68 total 50.01 last 145.26) - open/close 8621 (time 1746144765.68 total 60.01 last 120.60) - open/close 10000 (time 1746144775.26 total 69.59 last 143.98) - open/close 11468 (time 1746144785.26 total 79.59 last 146.73) - open/close 12973 (time 1746144795.27 total 89.60 last 150.37) - open/close 14273 (time 1746144805.27 total 99.60 last 129.98) - open/close 15437 (time 1746144815.28 total 109.61 last 116.32) - open/close 16718 (time 1746144825.28 total 119.61 last 128.06) - open/close 18020 (time 1746144835.29 total 129.62 last 130.11) - open/close 19466 (time 1746144845.31 total 139.64 last 144.34) - open/close 20000 (time 1746144849.23 total 143.56 last 136.04) - open/close 21383 (time 1746144859.24 total 153.57 last 138.20) - open/close 22542 (time 1746144869.25 total 163.58 last 115.83) - open/close 23755 (time 1746144879.25 total 173.58 last 121.28) - open/close 25074 (time 1746144889.25 total 183.58 last 131.88) - open/close 26244 (time 1746144899.25 total 193.58 last 116.95) - open/close 27746 (time 1746144909.26 total 203.59 last 150.14) - open/close 29040 (time 1746144919.26 total 213.59 last 129.36) - open/close 30000 (time 1746144926.75 total 221.07 last 128.28) - open/close 31244 (time 1746144936.75 total 231.08 last 124.38) - open/close 32565 (time 1746144946.75 total 241.08 last 132.06) - open/close 34015 (time 1746144956.75 total 251.08 last 144.97) - open/close 35338 (time 1746144966.76 total 261.09 last 132.23) - open/close 36602 (time 1746144976.76 total 271.09 last 126.35) - open/close 37892 (time 1746144986.76 total 281.09 last 128.96) - open/close 39186 (time 1746144996.77 total 291.10 last 129.35) - open/close 40000 (time 1746145003.20 total 297.53 last 126.55) - open/close 41356 (time 1746145013.21 total 307.54 last 135.51) - open/close 42665 (time 1746145023.21 total 317.54 last 130.85) - open/close 43933 (time 1746145033.21 total 327.54 last 126.76) - open/close 45154 (time 1746145043.21 total 337.54 last 122.10) - open/close 46355 (time 1746145053.22 total 347.55 last 120.03) - open/close 47593 (time 1746145063.22 total 357.55 last 123.78) - open/close 48843 (time 1746145073.23 total 367.55 last 124.96) - open/close 50000 (time 1746145082.05 total 376.38 last 131.13) - open/close 51635 (time 1746145092.05 total 386.38 last 163.46) - open/close 53142 (time 1746145102.05 total 396.38 last 150.68) - open/close 54490 (time 1746145112.06 total 406.39 last 134.72) - open/close 55791 (time 1746145122.06 total 416.39 last 130.09) - open/close 57267 (time 1746145132.06 total 426.39 last 147.53) - open/close 58747 (time 1746145142.07 total 436.40 last 147.93) - open/close 60000 (time 1746145150.02 total 444.35 last 157.56) - open/close 61365 (time 1746145160.02 total 454.35 last 136.48) - open/close 62810 (time 1746145170.03 total 464.36 last 144.35) - open/close 64014 (time 1746145180.04 total 474.37 last 120.32) total: 64768 open/close in 478.77 seconds: 135.28 ops/second - /unlink 3062 (time 1746145195.52 total 10.00 last 306.15) - /unlink 6309 (time 1746145205.53 total 20.00 last 324.60) - /unlink 9155 (time 1746145215.53 total 30.01 last 284.47) - /unlink 10000 (time 1746145219.59 total 34.06 last 208.35) - /unlink 12447 (time 1746145229.59 total 44.07 last 244.65) - /unlink 15074 (time 1746145239.59 total 54.07 last 262.65) - /unlink 17787 (time 1746145249.59 total 64.07 last 271.20) - /unlink 20000 (time 1746145259.35 total 73.83 last 226.84) - /unlink 22251 (time 1746145269.35 total 83.83 last 225.04) - /unlink 24776 (time 1746145279.36 total 93.84 last 252.37) - /unlink 26836 (time 1746145289.36 total 103.84 last 205.93) - /unlink 28862 (time 1746145299.36 total 113.84 last 202.52) - /unlink 30000 (time 1746145304.14 total 118.62 last 238.20) - /unlink 32162 (time 1746145314.15 total 128.63 last 216.01) - /unlink 34961 (time 1746145324.16 total 138.64 last 279.56) - /unlink 37803 (time 1746145334.17 total 148.65 last 284.07) - /unlink 40000 (time 1746145341.24 total 155.72 last 310.52) - /unlink 42457 (time 1746145351.24 total 165.72 last 245.68) - /unlink 45154 (time 1746145361.25 total 175.72 last 269.63) - /unlink 47653 (time 1746145371.25 total 185.73 last 249.88) - /unlink 50000 (time 1746145380.14 total 194.62 last 264.02) - /unlink 52536 (time 1746145390.14 total 204.62 last 253.59) - /unlink 55820 (time 1746145400.14 total 214.62 last 328.33) - /unlink 58480 (time 1746145410.16 total 224.64 last 265.54) - /unlink 60000 (time 1746145416.51 total 230.99 last 239.33) - /unlink 62537 (time 1746145426.51 total 240.99 last 253.69) total: 64768 /unlink in 250.22 seconds: 258.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1354 (time 1746145462.85 total 10.00 last 135.38) - open/close 2846 (time 1746145472.87 total 20.03 last 148.80) - open/close 4363 (time 1746145482.88 total 30.03 last 151.64) - open/close 5881 (time 1746145492.88 total 40.04 last 151.71) - open/close 7292 (time 1746145502.90 total 50.05 last 140.92) - open/close 8540 (time 1746145512.90 total 60.06 last 124.71) - open/close 9799 (time 1746145522.91 total 70.06 last 125.87) - open/close 10988 (time 1746145532.92 total 80.07 last 118.80) - open/close 12024 (time 1746145542.92 total 90.07 last 103.60) - open/close 13123 (time 1746145552.92 total 100.07 last 109.87) - open/close 14450 (time 1746145562.92 total 110.08 last 132.64) - open/close 15745 (time 1746145572.93 total 120.09 last 129.38) - open/close 17013 (time 1746145582.94 total 130.09 last 126.70) - open/close 18432 (time 1746145592.94 total 140.10 last 141.89) - open/close 19571 (time 1746145602.96 total 150.12 last 113.65) - open/close 20000 (time 1746145605.98 total 153.13 last 142.18) - open/close 21131 (time 1746145615.99 total 163.14 last 113.00) - open/close 22427 (time 1746145625.99 total 173.14 last 129.59) - open/close 23752 (time 1746145635.99 total 183.15 last 132.49) - open/close 25035 (time 1746145646.00 total 193.15 last 128.22) - open/close 26296 (time 1746145656.00 total 203.15 last 126.10) - open/close 27713 (time 1746145666.00 total 213.16 last 141.65) - open/close 28973 (time 1746145676.01 total 223.16 last 125.94) - open/close 30000 (time 1746145684.38 total 231.54 last 122.58) - open/close 31312 (time 1746145694.39 total 241.55 last 131.08) - open/close 32585 (time 1746145704.40 total 251.55 last 127.24) - open/close 34063 (time 1746145714.41 total 261.56 last 147.64) - open/close 35486 (time 1746145724.41 total 271.56 last 142.30) - open/close 36828 (time 1746145734.41 total 281.57 last 134.17) - open/close 38171 (time 1746145744.42 total 291.57 last 134.22) - open/close 39562 (time 1746145754.42 total 301.57 last 139.08) - open/close 40000 (time 1746145757.97 total 305.12 last 123.50) - open/close 41172 (time 1746145767.97 total 315.12 last 117.15) - open/close 42288 (time 1746145777.97 total 325.12 last 111.60) - open/close 43589 (time 1746145787.99 total 335.14 last 129.91) - open/close 44890 (time 1746145797.99 total 345.14 last 130.04) - open/close 46268 (time 1746145808.00 total 355.15 last 137.68) - open/close 47712 (time 1746145818.00 total 365.16 last 144.33) - open/close 49272 (time 1746145828.00 total 375.16 last 156.00) - open/close 50000 (time 1746145832.43 total 379.58 last 164.44) - open/close 51435 (time 1746145842.43 total 389.59 last 143.48) - open/close 52869 (time 1746145852.43 total 399.59 last 143.36) - open/close 54191 (time 1746145862.44 total 409.59 last 132.17) - open/close 55482 (time 1746145872.44 total 419.59 last 129.09) - open/close 56779 (time 1746145882.44 total 429.60 last 129.65) - open/close 57980 (time 1746145892.45 total 439.60 last 120.03) - open/close 59280 (time 1746145902.45 total 449.60 last 129.99) - open/close 60000 (time 1746145908.36 total 455.52 last 121.73) - open/close 61144 (time 1746145918.37 total 465.52 last 114.36) - open/close 62435 (time 1746145928.37 total 475.53 last 129.00) - open/close 63655 (time 1746145938.39 total 485.54 last 121.83) total: 64768 open/close in 494.88 seconds: 130.88 ops/second - /unlink 2350 (time 1746145959.48 total 10.00 last 234.95) - /unlink 4534 (time 1746145969.49 total 20.01 last 218.33) - /unlink 6656 (time 1746145979.49 total 30.01 last 212.17) - /unlink 8409 (time 1746145989.49 total 40.01 last 175.21) - /unlink 10000 (time 1746145995.97 total 46.49 last 245.62) - /unlink 12809 (time 1746146005.97 total 56.49 last 280.87) - /unlink 14990 (time 1746146015.97 total 66.49 last 218.09) - /unlink 17112 (time 1746146025.98 total 76.49 last 212.17) - /unlink 19265 (time 1746146035.99 total 86.50 last 215.06) - /unlink 20000 (time 1746146039.46 total 89.98 last 211.71) - /unlink 22736 (time 1746146049.46 total 99.98 last 273.58) - /unlink 25283 (time 1746146059.46 total 109.98 last 254.69) - /unlink 27886 (time 1746146069.46 total 119.98 last 260.28) - /unlink 30000 (time 1746146077.87 total 128.38 last 251.49) - /unlink 32584 (time 1746146087.87 total 138.38 last 258.39) - /unlink 34679 (time 1746146097.87 total 148.39 last 209.39) - /unlink 36815 (time 1746146107.87 total 158.39 last 213.53) - /unlink 39092 (time 1746146117.88 total 168.39 last 227.67) - /unlink 40000 (time 1746146121.56 total 172.08 last 246.63) - /unlink 42560 (time 1746146131.56 total 182.08 last 255.94) - /unlink 44883 (time 1746146141.56 total 192.08 last 232.29) - /unlink 47278 (time 1746146151.56 total 202.08 last 239.41) - /unlink 49437 (time 1746146161.57 total 212.09 last 215.76) - /unlink 50000 (time 1746146164.34 total 214.86 last 203.24) - /unlink 52052 (time 1746146174.35 total 224.86 last 205.08) - /unlink 54559 (time 1746146184.35 total 234.87 last 250.61) - /unlink 56810 (time 1746146194.35 total 244.87 last 225.04) - /unlink 60000 (time 1746146204.16 total 254.68 last 325.14) - /unlink 62586 (time 1746146214.16 total 264.68 last 258.58) total: 64768 /unlink in 273.73 seconds: 236.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1520 (time 1746146253.08 total 10.01 last 151.90) - open/close 2839 (time 1746146263.08 total 20.01 last 131.87) - open/close 4391 (time 1746146273.08 total 30.01 last 155.15) - open/close 6046 (time 1746146283.08 total 40.01 last 165.48) - open/close 7555 (time 1746146293.09 total 50.01 last 150.89) - open/close 9097 (time 1746146303.09 total 60.02 last 154.15) - open/close 10000 (time 1746146308.65 total 65.58 last 162.31) - open/close 11568 (time 1746146318.65 total 75.58 last 156.76) - open/close 13146 (time 1746146328.66 total 85.59 last 157.74) - open/close 14693 (time 1746146338.66 total 95.59 last 154.63) - open/close 16303 (time 1746146348.66 total 105.59 last 160.96) - open/close 17954 (time 1746146358.67 total 115.60 last 165.04) - open/close 19445 (time 1746146368.68 total 125.61 last 148.97) - open/close 20000 (time 1746146372.65 total 129.58 last 139.61) - open/close 21468 (time 1746146382.66 total 139.59 last 146.73) - open/close 22825 (time 1746146392.66 total 149.59 last 135.60) - open/close 24200 (time 1746146402.67 total 159.60 last 137.40) - open/close 25683 (time 1746146412.67 total 169.60 last 148.27) - open/close 26911 (time 1746146422.68 total 179.61 last 122.70) - open/close 28252 (time 1746146432.68 total 189.61 last 134.09) - open/close 29712 (time 1746146442.69 total 199.62 last 145.89) - open/close 30000 (time 1746146444.80 total 201.73 last 136.52) - open/close 31429 (time 1746146454.80 total 211.73 last 142.89) - open/close 32973 (time 1746146464.80 total 221.73 last 154.33) - open/close 34327 (time 1746146474.81 total 231.74 last 135.32) - open/close 35778 (time 1746146484.81 total 241.74 last 145.10) - open/close 37118 (time 1746146494.82 total 251.74 last 133.93) - open/close 38449 (time 1746146504.82 total 261.75 last 132.99) - open/close 39636 (time 1746146514.83 total 271.75 last 118.68) - open/close 40000 (time 1746146517.45 total 274.38 last 138.70) - open/close 41338 (time 1746146527.46 total 284.38 last 133.72) - open/close 42569 (time 1746146537.46 total 294.39 last 123.08) - open/close 43744 (time 1746146547.47 total 304.40 last 117.39) - open/close 44879 (time 1746146557.47 total 314.40 last 113.45) - open/close 46157 (time 1746146567.48 total 324.41 last 127.72) - open/close 47224 (time 1746146577.48 total 334.41 last 106.69) - open/close 48326 (time 1746146587.48 total 344.41 last 110.17) - open/close 49493 (time 1746146597.49 total 354.42 last 116.61) - open/close 50000 (time 1746146601.71 total 358.64 last 120.20) - open/close 51188 (time 1746146611.71 total 368.64 last 118.77) - open/close 52334 (time 1746146621.72 total 378.65 last 114.46) - open/close 53484 (time 1746146631.73 total 388.65 last 114.95) - open/close 54569 (time 1746146641.73 total 398.66 last 108.45) - open/close 55789 (time 1746146651.74 total 408.66 last 121.95) - open/close 56983 (time 1746146661.75 total 418.67 last 119.28) - open/close 58183 (time 1746146671.75 total 428.68 last 119.95) - open/close 59394 (time 1746146681.75 total 438.68 last 121.07) - open/close 60000 (time 1746146686.43 total 443.36 last 129.43) - open/close 61179 (time 1746146696.44 total 453.37 last 117.82) - open/close 62371 (time 1746146706.45 total 463.38 last 119.04) - open/close 63507 (time 1746146716.46 total 473.38 last 113.57) - open/close 64748 (time 1746146726.46 total 483.39 last 124.06) total: 64768 open/close in 483.57 seconds: 133.94 ops/second - /unlink 2452 (time 1746146738.16 total 10.00 last 245.16) - /unlink 5218 (time 1746146748.16 total 20.01 last 276.50) - /unlink 7710 (time 1746146758.17 total 30.01 last 249.06) - /unlink 10000 (time 1746146767.43 total 39.27 last 247.31) - /unlink 12317 (time 1746146777.43 total 49.27 last 231.65) - /unlink 14405 (time 1746146787.43 total 59.28 last 208.69) - /unlink 16510 (time 1746146797.44 total 69.28 last 210.44) - /unlink 18465 (time 1746146807.44 total 79.28 last 195.48) - /unlink 20000 (time 1746146814.97 total 86.81 last 203.77) - /unlink 21851 (time 1746146824.97 total 96.82 last 185.06) - /unlink 24119 (time 1746146834.97 total 106.82 last 226.80) - /unlink 26637 (time 1746146844.98 total 116.82 last 251.73) - /unlink 29112 (time 1746146854.98 total 126.82 last 247.42) - /unlink 30000 (time 1746146858.90 total 130.75 last 226.26) - /unlink 32689 (time 1746146868.90 total 140.75 last 268.87) - /unlink 35279 (time 1746146878.90 total 150.75 last 259.00) - /unlink 37971 (time 1746146888.91 total 160.75 last 269.17) - /unlink 40000 (time 1746146895.39 total 167.24 last 312.83) - /unlink 42725 (time 1746146905.40 total 177.24 last 272.37) - /unlink 45151 (time 1746146915.40 total 187.24 last 242.59) - /unlink 47466 (time 1746146925.40 total 197.24 last 231.42) - /unlink 49704 (time 1746146935.40 total 207.25 last 223.72) - /unlink 52254 (time 1746146945.41 total 217.26 last 254.81) - /unlink 54282 (time 1746146955.41 total 227.26 last 202.80) - /unlink 56267 (time 1746146965.41 total 237.26 last 198.44) - /unlink 58347 (time 1746146975.42 total 247.26 last 207.91) - /unlink 60000 (time 1746146981.96 total 253.80 last 252.87) - /unlink 62378 (time 1746146991.96 total 263.80 last 237.71) - /unlink 64646 (time 1746147001.96 total 273.81 last 226.76) total: 64768 /unlink in 274.36 seconds: 236.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1185 (time 1746147034.52 total 10.00 last 118.48) - open/close 2310 (time 1746147044.53 total 20.01 last 112.38) - open/close 3627 (time 1746147054.54 total 30.02 last 131.63) - open/close 4982 (time 1746147064.54 total 40.02 last 135.48) - open/close 6249 (time 1746147074.54 total 50.02 last 126.69) - open/close 7579 (time 1746147084.55 total 60.03 last 132.93) - open/close 8891 (time 1746147094.55 total 70.03 last 131.18) - open/close 10000 (time 1746147102.89 total 78.37 last 132.90) - open/close 11292 (time 1746147112.90 total 88.38 last 129.13) - open/close 12581 (time 1746147122.90 total 98.38 last 128.85) - open/close 13829 (time 1746147132.91 total 108.39 last 124.69) - open/close 15079 (time 1746147142.91 total 118.39 last 124.97) - open/close 16272 (time 1746147152.91 total 128.39 last 119.26) - open/close 17612 (time 1746147162.92 total 138.40 last 133.91) - open/close 18773 (time 1746147172.92 total 148.40 last 116.06) - open/close 20000 (time 1746147182.38 total 157.86 last 129.77) - open/close 21194 (time 1746147192.38 total 167.86 last 119.40) - open/close 22623 (time 1746147202.38 total 177.86 last 142.85) - open/close 24015 (time 1746147212.39 total 187.87 last 139.14) - open/close 25549 (time 1746147222.39 total 197.87 last 153.35) - open/close 26803 (time 1746147232.39 total 207.87 last 125.35) - open/close 28272 (time 1746147242.41 total 217.89 last 146.71) - open/close 29658 (time 1746147252.41 total 227.89 last 138.57) - open/close 30000 (time 1746147255.19 total 230.67 last 123.07) - open/close 31252 (time 1746147265.20 total 240.68 last 125.09) - open/close 32508 (time 1746147275.20 total 250.68 last 125.59) - open/close 33724 (time 1746147285.20 total 260.68 last 121.53) - open/close 34899 (time 1746147295.21 total 270.69 last 117.48) - open/close 36164 (time 1746147305.21 total 280.69 last 126.50) - open/close 37491 (time 1746147315.21 total 290.69 last 132.70) - open/close 38903 (time 1746147325.21 total 300.69 last 141.14) - open/close 40000 (time 1746147333.86 total 309.34 last 126.86) - open/close 41296 (time 1746147343.86 total 319.34 last 129.51) - open/close 42473 (time 1746147353.87 total 329.35 last 117.68) - open/close 43668 (time 1746147363.87 total 339.35 last 119.48) - open/close 44834 (time 1746147373.87 total 349.35 last 116.52) - open/close 46093 (time 1746147383.88 total 359.36 last 125.88) - open/close 47375 (time 1746147393.88 total 369.36 last 128.18) - open/close 48566 (time 1746147403.88 total 379.36 last 119.06) - open/close 49731 (time 1746147413.89 total 389.37 last 116.43) - open/close 50000 (time 1746147416.19 total 391.66 last 117.08) - open/close 51269 (time 1746147426.19 total 401.67 last 126.88) - open/close 52477 (time 1746147436.19 total 411.67 last 120.79) - open/close 53690 (time 1746147446.20 total 421.67 last 121.21) - open/close 54867 (time 1746147456.21 total 431.69 last 117.58) - open/close 56157 (time 1746147466.21 total 441.69 last 128.97) - open/close 57478 (time 1746147476.21 total 451.69 last 132.10) - open/close 58597 (time 1746147486.21 total 461.69 last 111.86) - open/close 59825 (time 1746147496.21 total 471.69 last 122.77) - open/close 61522 (time 1746147506.22 total 481.70 last 169.62) - open/close 62924 (time 1746147516.22 total 491.70 last 140.14) - open/close 64458 (time 1746147526.23 total 501.71 last 153.35) total: 64768 open/close in 504.20 seconds: 128.46 ops/second - /unlink 2342 (time 1746147540.28 total 10.00 last 234.14) - /unlink 4409 (time 1746147550.28 total 20.00 last 206.68) - /unlink 6853 (time 1746147560.29 total 30.01 last 244.27) - /unlink 9371 (time 1746147570.29 total 40.01 last 251.76) - /unlink 10000 (time 1746147572.64 total 42.36 last 267.48) - /unlink 12564 (time 1746147582.64 total 52.36 last 256.37) - /unlink 15145 (time 1746147592.65 total 62.37 last 257.99) - /unlink 17752 (time 1746147602.65 total 72.37 last 260.69) - /unlink 20000 (time 1746147611.54 total 81.27 last 252.64) - /unlink 23051 (time 1746147621.55 total 91.27 last 305.05) - /unlink 25128 (time 1746147631.55 total 101.27 last 207.70) - /unlink 27046 (time 1746147641.55 total 111.27 last 191.80) - /unlink 28934 (time 1746147651.55 total 121.27 last 188.73) - /unlink 30000 (time 1746147657.28 total 127.00 last 186.08) - /unlink 32052 (time 1746147667.28 total 137.00 last 205.18) - /unlink 34034 (time 1746147677.28 total 147.00 last 198.16) - /unlink 36157 (time 1746147687.29 total 157.01 last 212.20) - /unlink 38476 (time 1746147697.29 total 167.01 last 231.84) - /unlink 40000 (time 1746147704.00 total 173.73 last 226.95) - /unlink 42266 (time 1746147714.01 total 183.73 last 226.47) - /unlink 44350 (time 1746147724.01 total 193.73 last 208.35) - /unlink 46482 (time 1746147734.01 total 203.73 last 213.20) - /unlink 48924 (time 1746147744.01 total 213.73 last 244.17) - /unlink 50000 (time 1746147748.12 total 217.84 last 262.30) - /unlink 52070 (time 1746147758.12 total 227.84 last 206.95) - /unlink 54420 (time 1746147768.12 total 237.84 last 234.95) - /unlink 56665 (time 1746147778.12 total 247.84 last 224.47) - /unlink 58723 (time 1746147788.12 total 257.84 last 205.79) - /unlink 60000 (time 1746147794.51 total 264.23 last 199.98) - /unlink 62615 (time 1746147804.51 total 274.23 last 261.48) total: 64768 /unlink in 281.46 seconds: 230.11 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1577 (time 1746147841.97 total 10.00 last 157.68) - open/close 2855 (time 1746147851.98 total 20.01 last 127.75) - open/close 4040 (time 1746147861.98 total 30.01 last 118.49) - open/close 5454 (time 1746147871.98 total 40.01 last 141.35) - open/close 6846 (time 1746147881.99 total 50.02 last 138.99) - open/close 8210 (time 1746147892.00 total 60.03 last 136.33) - open/close 9496 (time 1746147902.00 total 70.03 last 128.54) - open/close 10000 (time 1746147905.75 total 73.78 last 134.44) - open/close 11312 (time 1746147915.76 total 83.79 last 131.07) - open/close 12665 (time 1746147925.77 total 93.80 last 135.17) - open/close 14087 (time 1746147935.78 total 103.81 last 142.14) - open/close 15549 (time 1746147945.78 total 113.81 last 146.11) - open/close 17173 (time 1746147955.78 total 123.82 last 162.38) - open/close 19180 (time 1746147965.79 total 133.82 last 200.64) - open/close 20000 (time 1746147971.15 total 139.18 last 152.80) - open/close 21571 (time 1746147981.16 total 149.19 last 156.98) - open/close 22975 (time 1746147991.16 total 159.19 last 140.40) - open/close 24405 (time 1746148001.17 total 169.20 last 142.93) - open/close 25877 (time 1746148011.17 total 179.20 last 147.19) - open/close 27516 (time 1746148021.17 total 189.20 last 163.90) - open/close 28911 (time 1746148031.17 total 199.20 last 139.44) - open/close 30000 (time 1746148038.59 total 206.62 last 146.76) - open/close 31896 (time 1746148048.60 total 216.63 last 189.53) - open/close 33928 (time 1746148058.60 total 226.63 last 203.09) - open/close 35454 (time 1746148068.61 total 236.64 last 152.56) - open/close 36959 (time 1746148078.61 total 246.64 last 150.45) - open/close 38331 (time 1746148088.61 total 256.64 last 137.16) - open/close 39819 (time 1746148098.61 total 266.64 last 148.79) - open/close 41069 (time 1746148108.62 total 276.65 last 124.91) - open/close 42404 (time 1746148118.63 total 286.66 last 133.39) - open/close 43957 (time 1746148128.63 total 296.66 last 155.24) - open/close 45730 (time 1746148138.64 total 306.67 last 177.18) - open/close 47277 (time 1746148148.65 total 316.68 last 154.49) - open/close 48822 (time 1746148158.66 total 326.69 last 154.43) - open/close 50000 (time 1746148166.35 total 334.38 last 153.05) - open/close 51314 (time 1746148176.36 total 344.39 last 131.37) - open/close 52627 (time 1746148186.36 total 354.39 last 131.23) - open/close 53956 (time 1746148196.36 total 364.39 last 132.86) - open/close 55387 (time 1746148206.37 total 374.40 last 143.01) - open/close 56918 (time 1746148216.37 total 384.40 last 153.08) - open/close 58415 (time 1746148226.38 total 394.41 last 149.56) - open/close 59961 (time 1746148236.39 total 404.42 last 154.53) - open/close 61207 (time 1746148246.39 total 414.42 last 124.57) - open/close 62633 (time 1746148256.39 total 424.42 last 142.54) - open/close 64124 (time 1746148266.40 total 434.43 last 149.01) total: 64768 open/close in 438.59 seconds: 147.67 ops/second - /unlink 2922 (time 1746148282.03 total 10.00 last 292.19) - /unlink 5639 (time 1746148292.03 total 20.00 last 271.70) - /unlink 8337 (time 1746148302.03 total 30.00 last 269.79) - /unlink 10000 (time 1746148309.43 total 37.41 last 224.57) - /unlink 12449 (time 1746148319.44 total 47.41 last 244.77) - /unlink 14777 (time 1746148329.44 total 57.41 last 232.76) - /unlink 17351 (time 1746148339.44 total 67.41 last 257.39) - /unlink 20000 (time 1746148349.00 total 76.98 last 276.98) - /unlink 22921 (time 1746148359.00 total 86.98 last 292.10) - /unlink 25474 (time 1746148369.01 total 96.98 last 255.24) - /unlink 28227 (time 1746148379.01 total 106.98 last 275.25) - /unlink 30000 (time 1746148385.59 total 113.56 last 269.44) - /unlink 32742 (time 1746148395.59 total 123.56 last 274.20) - /unlink 35708 (time 1746148405.59 total 133.56 last 296.59) - /unlink 37958 (time 1746148415.59 total 143.57 last 224.90) - /unlink 40000 (time 1746148424.15 total 152.12 last 238.66) - /unlink 42319 (time 1746148434.15 total 162.12 last 231.87) - /unlink 44314 (time 1746148444.15 total 172.13 last 199.46) - /unlink 46636 (time 1746148454.16 total 182.13 last 232.12) - /unlink 48751 (time 1746148464.16 total 192.13 last 211.42) - /unlink 50000 (time 1746148469.47 total 197.44 last 235.18) - /unlink 52101 (time 1746148479.47 total 207.45 last 210.03) - /unlink 54218 (time 1746148489.48 total 217.45 last 211.67) - /unlink 56400 (time 1746148499.48 total 227.45 last 218.16) - /unlink 58552 (time 1746148509.48 total 237.45 last 215.18) - /unlink 60000 (time 1746148516.02 total 243.99 last 221.42) - /unlink 62174 (time 1746148526.02 total 253.99 last 217.40) - /unlink 64250 (time 1746148536.02 total 263.99 last 207.57) total: 64768 /unlink in 266.40 seconds: 243.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server unloading modules via unload_modules_local on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2