== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:46:07 (1746135967) Stopping clients: oleg401-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg401-client.virtnet /mnt/lustre2 (opts:-f) oleg401-server: oleg401-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing load_modules_local oleg401-server: Loading modules from /home/green/git/lustre-release/lustre oleg401-server: detected 4 online CPUs by sysfs oleg401-server: MODOPTS_LIBCFS= oleg401-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg401-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg401-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1385 (time 1746136016.61 total 10.00 last 138.44) - open/close 2906 (time 1746136026.61 total 20.01 last 152.08) - open/close 4304 (time 1746136036.62 total 30.01 last 139.72) - open/close 5656 (time 1746136046.62 total 40.01 last 135.19) - open/close 6949 (time 1746136056.63 total 50.02 last 129.18) - open/close 8416 (time 1746136066.63 total 60.02 last 146.69) - open/close 9842 (time 1746136076.63 total 70.03 last 142.56) - open/close 11197 (time 1746136086.64 total 80.03 last 135.43) - open/close 12525 (time 1746136096.64 total 90.03 last 132.75) - open/close 14000 (time 1746136106.66 total 100.05 last 147.27) - open/close 15397 (time 1746136116.67 total 110.06 last 139.62) - open/close 16565 (time 1746136126.67 total 120.06 last 116.79) - open/close 17867 (time 1746136136.67 total 130.06 last 130.14) - open/close 19467 (time 1746136146.68 total 140.07 last 159.90) - open/close 20000 (time 1746136150.13 total 143.52 last 154.36) - open/close 21603 (time 1746136160.13 total 153.52 last 160.29) - open/close 23786 (time 1746136170.13 total 163.52 last 218.24) - open/close 25736 (time 1746136180.14 total 173.53 last 194.86) - open/close 27288 (time 1746136190.15 total 183.54 last 155.07) - open/close 28746 (time 1746136200.15 total 193.54 last 145.78) - open/close 29831 (time 1746136210.16 total 203.55 last 108.37) - open/close 31208 (time 1746136220.17 total 213.56 last 137.63) - open/close 32552 (time 1746136230.17 total 223.56 last 134.35) - open/close 33670 (time 1746136240.19 total 233.59 last 111.54) - open/close 34955 (time 1746136250.20 total 243.59 last 128.39) - open/close 36283 (time 1746136260.21 total 253.60 last 132.77) - open/close 37368 (time 1746136270.21 total 263.60 last 108.41) - open/close 38489 (time 1746136280.22 total 273.61 last 112.03) - open/close 39600 (time 1746136290.23 total 283.62 last 111.00) - open/close 40000 (time 1746136293.69 total 287.08 last 115.62) - open/close 41104 (time 1746136303.70 total 297.09 last 110.31) - open/close 42436 (time 1746136313.70 total 307.09 last 133.20) - open/close 43736 (time 1746136323.70 total 317.09 last 129.97) - open/close 44981 (time 1746136333.70 total 327.09 last 124.46) - open/close 46740 (time 1746136343.71 total 337.10 last 175.83) - open/close 47935 (time 1746136353.71 total 347.10 last 119.50) - open/close 49216 (time 1746136363.71 total 357.10 last 128.04) - open/close 50000 (time 1746136368.77 total 362.16 last 155.08) - open/close 51379 (time 1746136378.77 total 372.17 last 137.81) - open/close 52639 (time 1746136388.78 total 382.17 last 125.98) - open/close 53962 (time 1746136398.78 total 392.18 last 132.19) - open/close 55353 (time 1746136408.79 total 402.18 last 139.05) - open/close 56594 (time 1746136418.79 total 412.18 last 124.07) - open/close 57733 (time 1746136428.79 total 422.19 last 113.86) - open/close 58923 (time 1746136438.80 total 432.19 last 118.98) - open/close 60000 (time 1746136446.97 total 440.36 last 131.83) - open/close 61361 (time 1746136456.97 total 450.36 last 136.08) - open/close 62665 (time 1746136466.97 total 460.36 last 130.35) - open/close 63915 (time 1746136476.99 total 470.38 last 124.78) total: 64768 open/close in 477.54 seconds: 135.63 ops/second - /unlink 2190 (time 1746136496.05 total 10.00 last 218.94) - /unlink 4657 (time 1746136506.05 total 20.00 last 246.68) - /unlink 6955 (time 1746136516.05 total 30.00 last 229.78) - /unlink 9105 (time 1746136526.06 total 40.01 last 214.83) - /unlink 10000 (time 1746136530.07 total 44.02 last 223.42) - /unlink 12409 (time 1746136540.07 total 54.02 last 240.88) - /unlink 14461 (time 1746136550.07 total 64.02 last 205.08) - /unlink 16355 (time 1746136560.08 total 74.03 last 189.35) - /unlink 18190 (time 1746136570.08 total 84.03 last 183.39) - /unlink 20000 (time 1746136579.01 total 92.96 last 202.82) - /unlink 21861 (time 1746136589.01 total 102.96 last 186.03) - /unlink 24121 (time 1746136599.01 total 112.96 last 225.96) - /unlink 26116 (time 1746136609.01 total 122.96 last 199.47) - /unlink 27992 (time 1746136619.01 total 132.97 last 187.58) - /unlink 30000 (time 1746136628.50 total 142.45 last 211.75) - /unlink 31804 (time 1746136638.50 total 152.45 last 180.33) - /unlink 33563 (time 1746136648.50 total 162.46 last 175.85) - /unlink 35761 (time 1746136658.51 total 172.47 last 219.57) - /unlink 38028 (time 1746136668.52 total 182.47 last 226.68) - /unlink 40000 (time 1746136677.80 total 191.75 last 212.39) - /unlink 42326 (time 1746136687.80 total 201.75 last 232.59) - /unlink 44974 (time 1746136697.80 total 211.75 last 264.71) - /unlink 47476 (time 1746136707.81 total 221.76 last 250.07) - /unlink 49761 (time 1746136717.82 total 231.77 last 228.35) - /unlink 51856 (time 1746136727.82 total 241.77 last 209.47) - /unlink 54321 (time 1746136737.82 total 251.77 last 246.44) - /unlink 56756 (time 1746136747.82 total 261.77 last 243.41) - /unlink 59462 (time 1746136757.82 total 271.78 last 270.53) - /unlink 62113 (time 1746136767.83 total 281.78 last 265.08) total: 64768 /unlink in 291.19 seconds: 222.43 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1690 (time 1746136803.90 total 10.00 last 168.96) - open/close 3129 (time 1746136813.90 total 20.00 last 143.87) - open/close 4533 (time 1746136823.90 total 30.01 last 140.38) - open/close 5885 (time 1746136833.91 total 40.01 last 135.14) - open/close 7166 (time 1746136843.91 total 50.01 last 128.06) - open/close 8396 (time 1746136853.92 total 60.02 last 122.92) - open/close 9490 (time 1746136863.93 total 70.04 last 109.22) - open/close 10000 (time 1746136869.01 total 75.12 last 100.36) - open/close 11388 (time 1746136879.02 total 85.12 last 138.77) - open/close 12693 (time 1746136889.02 total 95.12 last 130.49) - open/close 13991 (time 1746136899.02 total 105.12 last 129.75) - open/close 15225 (time 1746136909.02 total 115.13 last 123.34) - open/close 16324 (time 1746136919.03 total 125.14 last 109.84) - open/close 17762 (time 1746136929.04 total 135.14 last 143.70) - open/close 19083 (time 1746136939.04 total 145.14 last 132.09) - open/close 20000 (time 1746136944.97 total 151.08 last 154.55) - open/close 21418 (time 1746136954.97 total 161.08 last 141.77) - open/close 22715 (time 1746136964.98 total 171.08 last 129.64) - open/close 23945 (time 1746136974.98 total 181.08 last 122.99) - open/close 25235 (time 1746136984.99 total 191.09 last 128.93) - open/close 26417 (time 1746136995.00 total 201.10 last 118.07) - open/close 27541 (time 1746137005.00 total 211.10 last 112.39) - open/close 28639 (time 1746137015.00 total 221.11 last 109.74) - open/close 30000 (time 1746137024.73 total 230.84 last 139.90) - open/close 31374 (time 1746137034.73 total 240.84 last 137.36) - open/close 32768 (time 1746137044.74 total 250.84 last 139.38) - open/close 34270 (time 1746137054.74 total 260.84 last 150.13) - open/close 35729 (time 1746137064.75 total 270.85 last 145.83) - open/close 37151 (time 1746137074.75 total 280.85 last 142.18) - open/close 38689 (time 1746137084.75 total 290.85 last 153.79) - open/close 40000 (time 1746137094.13 total 300.23 last 139.75) - open/close 41428 (time 1746137104.13 total 310.24 last 142.73) - open/close 42726 (time 1746137114.14 total 320.25 last 129.66) - open/close 43948 (time 1746137124.15 total 330.25 last 122.17) - open/close 45169 (time 1746137134.15 total 340.26 last 122.05) - open/close 46519 (time 1746137144.16 total 350.26 last 134.94) - open/close 47870 (time 1746137154.16 total 360.26 last 135.05) - open/close 49116 (time 1746137164.16 total 370.26 last 124.59) - open/close 50000 (time 1746137171.38 total 377.49 last 122.36) - open/close 51452 (time 1746137181.39 total 387.49 last 145.19) - open/close 53228 (time 1746137191.39 total 397.49 last 177.54) - open/close 54875 (time 1746137201.39 total 407.50 last 164.60) - open/close 56162 (time 1746137211.40 total 417.50 last 128.65) - open/close 57756 (time 1746137221.40 total 427.51 last 159.31) - open/close 59243 (time 1746137231.40 total 437.51 last 148.70) - open/close 60000 (time 1746137237.14 total 443.24 last 132.07) - open/close 61549 (time 1746137247.14 total 453.25 last 154.83) - open/close 62995 (time 1746137257.14 total 463.25 last 144.58) - open/close 64656 (time 1746137267.14 total 473.25 last 166.08) total: 64768 open/close in 473.86 seconds: 136.68 ops/second - /unlink 3073 (time 1746137278.93 total 10.00 last 307.24) - /unlink 5610 (time 1746137288.93 total 20.00 last 253.65) - /unlink 8638 (time 1746137298.94 total 30.01 last 302.75) - /unlink 10000 (time 1746137303.39 total 34.46 last 305.87) - /unlink 12630 (time 1746137313.39 total 44.46 last 262.96) - /unlink 15155 (time 1746137323.39 total 54.46 last 252.44) - /unlink 17960 (time 1746137333.39 total 64.46 last 280.50) - /unlink 20000 (time 1746137340.71 total 71.78 last 278.65) - /unlink 22643 (time 1746137350.72 total 81.79 last 264.14) - /unlink 25203 (time 1746137360.73 total 91.80 last 255.76) - /unlink 27987 (time 1746137370.73 total 101.80 last 278.28) - /unlink 30000 (time 1746137378.39 total 109.46 last 262.90) - /unlink 32840 (time 1746137388.39 total 119.46 last 283.99) - /unlink 35462 (time 1746137398.39 total 129.46 last 262.19) - /unlink 38059 (time 1746137408.39 total 139.46 last 259.68) - /unlink 40000 (time 1746137415.34 total 146.41 last 279.50) - /unlink 42915 (time 1746137425.34 total 156.41 last 291.33) - /unlink 45310 (time 1746137435.35 total 166.42 last 239.35) - /unlink 47758 (time 1746137445.35 total 176.42 last 244.74) - /unlink 50000 (time 1746137454.35 total 185.41 last 249.29) - /unlink 52491 (time 1746137464.35 total 195.42 last 249.05) - /unlink 55072 (time 1746137474.35 total 205.42 last 258.08) - /unlink 57568 (time 1746137484.35 total 215.42 last 249.60) - /unlink 60000 (time 1746137493.01 total 224.08 last 280.62) - /unlink 62678 (time 1746137503.01 total 234.08 last 267.78) total: 64768 /unlink in 240.20 seconds: 269.64 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1807 (time 1746137539.95 total 10.00 last 180.68) - open/close 3367 (time 1746137549.96 total 20.00 last 155.96) - open/close 4719 (time 1746137559.96 total 30.01 last 135.16) - open/close 6113 (time 1746137569.97 total 40.01 last 139.31) - open/close 7557 (time 1746137579.97 total 50.02 last 144.35) - open/close 9122 (time 1746137589.98 total 60.02 last 156.42) - open/close 10000 (time 1746137595.46 total 65.50 last 160.17) - open/close 11372 (time 1746137605.46 total 75.51 last 137.14) - open/close 12928 (time 1746137615.46 total 85.51 last 155.58) - open/close 14379 (time 1746137625.47 total 95.51 last 145.04) - open/close 15977 (time 1746137635.47 total 105.52 last 159.71) - open/close 17342 (time 1746137645.48 total 115.53 last 136.40) - open/close 18749 (time 1746137655.48 total 125.53 last 140.69) - open/close 20000 (time 1746137663.89 total 133.93 last 148.81) - open/close 21179 (time 1746137673.89 total 143.94 last 117.85) - open/close 22588 (time 1746137683.90 total 153.94 last 140.81) - open/close 24067 (time 1746137693.90 total 163.95 last 147.89) - open/close 25695 (time 1746137703.91 total 173.95 last 162.66) - open/close 27286 (time 1746137713.91 total 183.96 last 159.08) - open/close 28685 (time 1746137723.91 total 193.96 last 139.87) - open/close 30000 (time 1746137731.74 total 201.79 last 167.89) - open/close 31491 (time 1746137741.75 total 211.80 last 149.01) - open/close 32926 (time 1746137751.76 total 221.80 last 143.37) - open/close 34767 (time 1746137761.76 total 231.81 last 184.07) - open/close 36268 (time 1746137771.76 total 241.81 last 150.08) - open/close 37850 (time 1746137781.76 total 251.81 last 158.16) - open/close 39425 (time 1746137791.77 total 261.82 last 157.41) - open/close 40000 (time 1746137794.39 total 264.44 last 219.11) - open/close 41595 (time 1746137804.40 total 274.44 last 159.44) - open/close 42986 (time 1746137814.40 total 284.45 last 139.03) - open/close 44307 (time 1746137824.41 total 294.46 last 132.00) - open/close 45679 (time 1746137834.41 total 304.46 last 137.19) - open/close 47123 (time 1746137844.42 total 314.47 last 144.28) - open/close 48808 (time 1746137854.42 total 324.47 last 168.49) - open/close 50000 (time 1746137861.89 total 331.94 last 159.58) - open/close 51351 (time 1746137871.89 total 341.94 last 135.09) - open/close 52757 (time 1746137881.90 total 351.95 last 140.45) - open/close 54310 (time 1746137891.90 total 361.95 last 155.23) - open/close 55806 (time 1746137901.91 total 371.95 last 149.58) - open/close 57262 (time 1746137911.91 total 381.95 last 145.59) - open/close 58927 (time 1746137921.91 total 391.96 last 166.46) - open/close 60000 (time 1746137928.02 total 398.06 last 175.73) - open/close 61905 (time 1746137938.02 total 408.06 last 190.46) - open/close 63662 (time 1746137948.02 total 418.07 last 175.65) total: 64768 open/close in 423.45 seconds: 152.95 ops/second - /unlink 3181 (time 1746137964.62 total 10.00 last 318.06) - /unlink 7115 (time 1746137974.62 total 20.00 last 393.38) - /unlink 10000 (time 1746137982.49 total 27.87 last 366.68) - /unlink 13315 (time 1746137992.49 total 37.87 last 331.47) - /unlink 16179 (time 1746138002.49 total 47.88 last 286.25) - /unlink 19374 (time 1746138012.50 total 57.88 last 319.47) - /unlink 22316 (time 1746138022.50 total 67.88 last 294.15) - /unlink 25966 (time 1746138032.50 total 77.88 last 364.89) - /unlink 28605 (time 1746138042.50 total 87.88 last 263.82) - /unlink 30000 (time 1746138048.34 total 93.72 last 238.90) - /unlink 32978 (time 1746138058.34 total 103.72 last 297.76) - /unlink 35390 (time 1746138068.35 total 113.73 last 241.15) - /unlink 37718 (time 1746138078.35 total 123.73 last 232.80) - /unlink 40000 (time 1746138084.95 total 130.33 last 345.72) - /unlink 42930 (time 1746138094.95 total 140.33 last 292.98) - /unlink 45871 (time 1746138104.95 total 150.33 last 294.08) - /unlink 49238 (time 1746138114.95 total 160.33 last 336.58) - /unlink 50000 (time 1746138117.90 total 163.28 last 258.66) - /unlink 53402 (time 1746138127.90 total 173.29 last 339.95) - /unlink 57339 (time 1746138137.91 total 183.29 last 393.64) - /unlink 60000 (time 1746138147.72 total 193.10 last 271.18) - /unlink 62869 (time 1746138157.72 total 203.10 last 286.88) total: 64768 /unlink in 210.00 seconds: 308.42 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1481 (time 1746138195.89 total 10.00 last 148.04) - open/close 3003 (time 1746138205.89 total 20.01 last 152.17) - open/close 4714 (time 1746138215.89 total 30.01 last 171.02) - open/close 6451 (time 1746138225.90 total 40.01 last 173.63) - open/close 8480 (time 1746138235.90 total 50.02 last 202.87) - open/close 10000 (time 1746138242.77 total 56.89 last 221.27) - open/close 12021 (time 1746138252.77 total 66.89 last 202.03) - open/close 13830 (time 1746138262.77 total 76.89 last 180.87) - open/close 15666 (time 1746138272.78 total 86.89 last 183.54) - open/close 17396 (time 1746138282.78 total 96.90 last 172.96) - open/close 19064 (time 1746138292.79 total 106.90 last 166.68) - open/close 20000 (time 1746138299.79 total 113.91 last 133.65) - open/close 21378 (time 1746138309.79 total 123.91 last 137.79) - open/close 22796 (time 1746138319.80 total 133.91 last 141.72) - open/close 24134 (time 1746138329.80 total 143.92 last 133.70) - open/close 25344 (time 1746138339.81 total 153.92 last 120.97) - open/close 26816 (time 1746138349.81 total 163.93 last 147.16) - open/close 28452 (time 1746138359.81 total 173.93 last 163.58) - open/close 30000 (time 1746138369.50 total 183.62 last 159.71) - open/close 31403 (time 1746138379.51 total 193.63 last 140.22) - open/close 32978 (time 1746138389.51 total 203.63 last 157.47) - open/close 34824 (time 1746138399.51 total 213.63 last 184.58) - open/close 36774 (time 1746138409.51 total 223.63 last 194.97) - open/close 38707 (time 1746138419.51 total 233.63 last 193.26) - open/close 40000 (time 1746138427.61 total 241.73 last 159.74) - open/close 41865 (time 1746138437.61 total 251.73 last 186.48) - open/close 43696 (time 1746138447.61 total 261.73 last 183.08) - open/close 45380 (time 1746138457.62 total 271.73 last 168.32) - open/close 46991 (time 1746138467.62 total 281.74 last 161.06) - open/close 48900 (time 1746138477.62 total 291.74 last 190.81) - open/close 50000 (time 1746138485.63 total 299.75 last 137.30) - open/close 51436 (time 1746138495.64 total 309.75 last 143.57) - open/close 52853 (time 1746138505.64 total 319.76 last 141.64) - open/close 54203 (time 1746138515.64 total 329.76 last 134.99) - open/close 55617 (time 1746138525.65 total 339.76 last 141.32) - open/close 56959 (time 1746138535.65 total 349.77 last 134.18) - open/close 58291 (time 1746138545.65 total 359.77 last 133.13) - open/close 59686 (time 1746138555.66 total 369.78 last 139.39) - open/close 60000 (time 1746138558.40 total 372.52 last 114.54) - open/close 61384 (time 1746138568.41 total 382.52 last 138.35) - open/close 62977 (time 1746138578.42 total 392.54 last 159.07) - open/close 64258 (time 1746138588.43 total 402.54 last 128.01) total: 64768 open/close in 406.04 seconds: 159.51 ops/second - /unlink 2101 (time 1746138603.41 total 10.00 last 210.06) - /unlink 4639 (time 1746138613.41 total 20.01 last 253.71) - /unlink 6676 (time 1746138623.42 total 30.01 last 203.67) - /unlink 9659 (time 1746138633.42 total 40.01 last 298.11) - /unlink 12096 (time 1746138643.43 total 50.02 last 243.61) - /unlink 15590 (time 1746138653.43 total 60.02 last 349.29) - /unlink 18260 (time 1746138663.43 total 70.02 last 266.97) - /unlink 20000 (time 1746138670.64 total 77.23 last 241.37) - /unlink 22866 (time 1746138680.64 total 87.23 last 286.54) - /unlink 25123 (time 1746138690.64 total 97.23 last 225.65) - /unlink 27336 (time 1746138700.64 total 107.23 last 221.29) - /unlink 29496 (time 1746138710.65 total 117.24 last 215.94) - /unlink 30000 (time 1746138713.37 total 119.96 last 184.81) - /unlink 31754 (time 1746138723.38 total 129.97 last 175.34) - /unlink 33735 (time 1746138733.38 total 139.97 last 198.06) - /unlink 36091 (time 1746138743.38 total 149.97 last 235.58) - /unlink 38396 (time 1746138753.38 total 159.97 last 230.49) - /unlink 40000 (time 1746138760.20 total 166.79 last 235.20) - /unlink 42270 (time 1746138770.20 total 176.79 last 226.93) - /unlink 44708 (time 1746138780.20 total 186.80 last 243.77) - /unlink 47102 (time 1746138790.21 total 196.80 last 239.33) - /unlink 49225 (time 1746138800.21 total 206.80 last 212.22) - /unlink 50000 (time 1746138804.13 total 210.72 last 197.78) - /unlink 52249 (time 1746138814.13 total 220.72 last 224.84) - /unlink 54369 (time 1746138824.13 total 230.72 last 211.97) - /unlink 56410 (time 1746138834.14 total 240.73 last 204.07) - /unlink 58753 (time 1746138844.14 total 250.73 last 234.26) - /unlink 60000 (time 1746138849.31 total 255.90 last 241.31) - /unlink 62152 (time 1746138859.31 total 265.90 last 215.18) - /unlink 64447 (time 1746138869.31 total 275.90 last 229.40) total: 64768 /unlink in 277.63 seconds: 233.29 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1176 (time 1746138903.03 total 10.01 last 117.49) - open/close 2361 (time 1746138913.03 total 20.01 last 118.48) - open/close 3473 (time 1746138923.03 total 30.01 last 111.17) - open/close 4646 (time 1746138933.04 total 40.02 last 117.25) - open/close 6036 (time 1746138943.05 total 50.03 last 138.84) - open/close 7385 (time 1746138953.05 total 60.03 last 134.84) - open/close 8762 (time 1746138963.06 total 70.04 last 137.58) - open/close 10000 (time 1746138972.70 total 79.68 last 128.41) - open/close 11047 (time 1746138982.70 total 89.68 last 104.69) - open/close 12093 (time 1746138992.71 total 99.69 last 104.51) - open/close 13262 (time 1746139002.72 total 109.70 last 116.82) - open/close 14518 (time 1746139012.72 total 119.70 last 125.57) - open/close 15746 (time 1746139022.72 total 129.71 last 122.75) - open/close 17188 (time 1746139032.72 total 139.71 last 144.19) - open/close 18486 (time 1746139042.73 total 149.71 last 129.77) - open/close 19702 (time 1746139052.73 total 159.72 last 121.50) - open/close 20000 (time 1746139055.38 total 162.36 last 112.84) - open/close 21132 (time 1746139065.38 total 172.36 last 113.14) - open/close 22252 (time 1746139075.38 total 182.36 last 111.98) - open/close 23375 (time 1746139085.39 total 192.37 last 112.27) - open/close 24861 (time 1746139095.39 total 202.37 last 148.56) - open/close 26178 (time 1746139105.39 total 212.38 last 131.63) - open/close 27656 (time 1746139115.40 total 222.38 last 147.74) - open/close 29645 (time 1746139125.40 total 232.38 last 198.85) - open/close 31218 (time 1746139135.41 total 242.39 last 157.18) - open/close 32467 (time 1746139145.42 total 252.40 last 124.78) - open/close 33851 (time 1746139155.42 total 262.40 last 138.36) - open/close 35348 (time 1746139165.43 total 272.41 last 149.59) - open/close 36779 (time 1746139175.43 total 282.41 last 143.05) - open/close 37927 (time 1746139185.43 total 292.42 last 114.76) - open/close 39162 (time 1746139195.43 total 302.42 last 123.50) - open/close 40000 (time 1746139201.41 total 308.39 last 140.30) - open/close 41251 (time 1746139211.41 total 318.39 last 125.09) - open/close 42462 (time 1746139221.41 total 328.40 last 121.01) - open/close 43778 (time 1746139231.42 total 338.40 last 131.57) - open/close 45120 (time 1746139241.42 total 348.40 last 134.20) - open/close 46558 (time 1746139251.42 total 358.40 last 143.74) - open/close 47737 (time 1746139261.43 total 368.42 last 117.77) - open/close 49184 (time 1746139271.44 total 378.42 last 144.63) - open/close 50000 (time 1746139277.02 total 384.00 last 146.29) - open/close 51627 (time 1746139287.02 total 394.00 last 162.64) - open/close 52827 (time 1746139297.07 total 404.05 last 119.41) - open/close 53857 (time 1746139307.07 total 414.06 last 102.96) - open/close 54960 (time 1746139317.08 total 424.06 last 110.23) - open/close 56302 (time 1746139327.08 total 434.06 last 134.16) - open/close 57738 (time 1746139337.08 total 444.07 last 143.59) - open/close 58991 (time 1746139347.09 total 454.07 last 125.25) - open/close 60000 (time 1746139355.43 total 462.41 last 120.92) - open/close 61143 (time 1746139365.44 total 472.42 last 114.23) - open/close 62241 (time 1746139375.44 total 482.42 last 109.79) - open/close 63462 (time 1746139385.44 total 492.42 last 122.05) - open/close 64690 (time 1746139395.45 total 502.43 last 122.75) total: 64768 open/close in 502.98 seconds: 128.77 ops/second - /unlink 2407 (time 1746139407.62 total 10.00 last 240.67) - /unlink 4671 (time 1746139417.62 total 20.00 last 226.34) - /unlink 6795 (time 1746139427.62 total 30.01 last 212.35) - /unlink 8658 (time 1746139437.63 total 40.01 last 186.21) - /unlink 10000 (time 1746139445.31 total 47.69 last 174.77) - /unlink 11728 (time 1746139455.32 total 57.70 last 172.64) - /unlink 13471 (time 1746139465.32 total 67.70 last 174.23) - /unlink 15379 (time 1746139475.32 total 77.70 last 190.78) - /unlink 17017 (time 1746139485.32 total 87.70 last 163.79) - /unlink 18674 (time 1746139495.32 total 97.70 last 165.70) - /unlink 20000 (time 1746139503.02 total 105.41 last 172.18) - /unlink 21882 (time 1746139513.02 total 115.41 last 188.16) - /unlink 24106 (time 1746139523.03 total 125.41 last 222.37) - /unlink 26186 (time 1746139533.03 total 135.41 last 207.93) - /unlink 28327 (time 1746139543.03 total 145.42 last 213.97) - /unlink 30000 (time 1746139550.86 total 153.24 last 213.84) - /unlink 32797 (time 1746139560.86 total 163.24 last 279.67) - /unlink 35215 (time 1746139570.86 total 173.24 last 241.76) - /unlink 37735 (time 1746139580.86 total 183.25 last 251.94) - /unlink 40000 (time 1746139587.89 total 190.28 last 322.19) - /unlink 42246 (time 1746139597.90 total 200.28 last 224.54) - /unlink 43787 (time 1746139607.90 total 210.28 last 154.08) - /unlink 45923 (time 1746139617.90 total 220.29 last 213.44) - /unlink 48314 (time 1746139627.91 total 230.29 last 238.94) - /unlink 50000 (time 1746139636.07 total 238.45 last 206.68) - /unlink 51906 (time 1746139646.07 total 248.46 last 190.54) - /unlink 54235 (time 1746139656.07 total 258.46 last 232.86) - /unlink 56394 (time 1746139666.08 total 268.46 last 215.81) - /unlink 58542 (time 1746139676.08 total 278.46 last 214.71) - /unlink 60000 (time 1746139682.27 total 284.66 last 235.43) - /unlink 62110 (time 1746139692.28 total 294.66 last 210.89) - /unlink 64142 (time 1746139702.28 total 304.67 last 203.11) total: 64768 /unlink in 307.85 seconds: 210.39 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg401-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg401-server unloading modules via unload_modules_local on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2