== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:33:53 (1746131633) Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:-f) oleg438-server: oleg438-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: MODOPTS_LIBCFS= oleg438-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1230 (time 1746131683.87 total 10.00 last 122.99) - open/close 2492 (time 1746131693.87 total 20.00 last 126.20) - open/close 3715 (time 1746131703.88 total 30.01 last 122.15) - open/close 4818 (time 1746131713.89 total 40.02 last 110.23) - open/close 5970 (time 1746131723.89 total 50.02 last 115.16) - open/close 7295 (time 1746131733.89 total 60.03 last 132.44) - open/close 8431 (time 1746131743.90 total 70.03 last 113.52) - open/close 9452 (time 1746131753.91 total 80.04 last 102.05) - open/close 10000 (time 1746131758.73 total 84.86 last 113.70) - open/close 11228 (time 1746131768.73 total 94.87 last 122.71) - open/close 12387 (time 1746131778.74 total 104.87 last 115.82) - open/close 13390 (time 1746131788.74 total 114.87 last 100.29) - open/close 14537 (time 1746131798.74 total 124.88 last 114.68) - open/close 15529 (time 1746131808.74 total 134.88 last 99.19) - open/close 16607 (time 1746131818.75 total 144.88 last 107.78) - open/close 17841 (time 1746131828.75 total 154.88 last 123.34) - open/close 18913 (time 1746131838.76 total 164.89 last 107.11) - open/close 19937 (time 1746131848.77 total 174.90 last 102.35) - open/close 21011 (time 1746131858.77 total 184.90 last 107.38) - open/close 22467 (time 1746131868.77 total 194.90 last 145.56) - open/close 23929 (time 1746131878.78 total 204.91 last 146.11) - open/close 25175 (time 1746131888.79 total 214.92 last 124.49) - open/close 26190 (time 1746131898.79 total 224.92 last 101.44) - open/close 27149 (time 1746131908.80 total 234.93 last 95.83) - open/close 28349 (time 1746131918.80 total 244.94 last 119.96) - open/close 29642 (time 1746131928.80 total 254.94 last 129.28) - open/close 30000 (time 1746131932.51 total 258.64 last 96.63) - open/close 31117 (time 1746131942.52 total 268.66 last 111.52) - open/close 32426 (time 1746131952.53 total 278.66 last 130.82) - open/close 33608 (time 1746131962.53 total 288.67 last 118.14) - open/close 35026 (time 1746131972.54 total 298.67 last 141.79) - open/close 36401 (time 1746131982.54 total 308.67 last 137.44) - open/close 37772 (time 1746131992.54 total 318.67 last 137.08) - open/close 38807 (time 1746132002.55 total 328.68 last 103.46) - open/close 39757 (time 1746132012.55 total 338.68 last 94.99) - open/close 40000 (time 1746132014.71 total 340.85 last 112.12) - open/close 41120 (time 1746132024.72 total 350.85 last 111.96) - open/close 42323 (time 1746132034.72 total 360.85 last 120.25) - open/close 43562 (time 1746132044.72 total 370.86 last 123.88) - open/close 44822 (time 1746132054.72 total 380.86 last 125.99) - open/close 46302 (time 1746132064.73 total 390.86 last 147.97) - open/close 47558 (time 1746132074.73 total 400.86 last 125.57) - open/close 48698 (time 1746132084.73 total 410.87 last 113.94) - open/close 49733 (time 1746132094.74 total 420.87 last 103.47) - open/close 50000 (time 1746132097.54 total 423.67 last 95.31) - open/close 51080 (time 1746132107.54 total 433.67 last 107.99) - open/close 52139 (time 1746132117.54 total 443.67 last 105.90) - open/close 53232 (time 1746132127.54 total 453.68 last 109.27) - open/close 54455 (time 1746132137.55 total 463.68 last 122.22) - open/close 55596 (time 1746132147.56 total 473.69 last 113.99) - open/close 56821 (time 1746132157.57 total 483.70 last 122.35) - open/close 58018 (time 1746132167.57 total 493.71 last 119.67) - open/close 59215 (time 1746132177.58 total 503.71 last 119.66) - open/close 60000 (time 1746132183.36 total 509.50 last 135.65) - open/close 61038 (time 1746132193.37 total 519.50 last 103.75) - open/close 61977 (time 1746132203.37 total 529.50 last 93.86) - open/close 62977 (time 1746132213.37 total 539.50 last 100.00) - open/close 64146 (time 1746132223.38 total 549.51 last 116.82) total: 64768 open/close in 555.12 seconds: 116.67 ops/second - /unlink 1687 (time 1746132240.76 total 10.00 last 168.69) - /unlink 3428 (time 1746132250.76 total 20.00 last 174.10) - /unlink 5127 (time 1746132260.77 total 30.00 last 169.87) - /unlink 6819 (time 1746132270.77 total 40.01 last 169.08) - /unlink 8622 (time 1746132280.78 total 50.02 last 180.19) - /unlink 10000 (time 1746132288.45 total 57.69 last 179.54) - /unlink 11757 (time 1746132298.45 total 67.69 last 175.67) - /unlink 13214 (time 1746132308.46 total 77.70 last 145.60) - /unlink 14742 (time 1746132318.47 total 87.71 last 152.66) - /unlink 16317 (time 1746132328.47 total 97.71 last 157.46) - /unlink 17861 (time 1746132338.48 total 107.71 last 154.35) - /unlink 19295 (time 1746132348.48 total 117.72 last 143.30) - /unlink 20000 (time 1746132353.34 total 122.58 last 145.22) - /unlink 21524 (time 1746132363.34 total 132.58 last 152.34) - /unlink 23450 (time 1746132373.34 total 142.58 last 192.56) - /unlink 25354 (time 1746132383.35 total 152.58 last 190.34) - /unlink 27313 (time 1746132393.35 total 162.59 last 195.88) - /unlink 29359 (time 1746132403.35 total 172.59 last 204.53) - /unlink 30000 (time 1746132406.27 total 175.51 last 219.51) - /unlink 32088 (time 1746132416.27 total 185.51 last 208.75) - /unlink 33974 (time 1746132426.28 total 195.51 last 188.53) - /unlink 36069 (time 1746132436.28 total 205.52 last 209.47) - /unlink 38722 (time 1746132446.28 total 215.52 last 265.28) - /unlink 40000 (time 1746132451.21 total 220.44 last 259.36) - /unlink 42536 (time 1746132461.21 total 230.45 last 253.54) - /unlink 44856 (time 1746132471.21 total 240.45 last 231.94) - /unlink 47146 (time 1746132481.21 total 250.45 last 228.99) - /unlink 49448 (time 1746132491.21 total 260.45 last 230.18) - /unlink 50000 (time 1746132493.53 total 262.77 last 238.03) - /unlink 52070 (time 1746132503.53 total 272.77 last 206.97) - /unlink 54135 (time 1746132513.53 total 282.77 last 206.49) - /unlink 56276 (time 1746132523.54 total 292.77 last 214.08) - /unlink 58564 (time 1746132533.54 total 302.78 last 228.75) - /unlink 60000 (time 1746132541.25 total 310.49 last 186.16) - /unlink 61862 (time 1746132551.25 total 320.49 last 186.15) - /unlink 64113 (time 1746132561.26 total 330.50 last 224.95) total: 64768 /unlink in 334.20 seconds: 193.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1312 (time 1746132595.65 total 10.01 last 131.10) - open/close 2545 (time 1746132605.66 total 20.01 last 123.22) - open/close 3763 (time 1746132615.66 total 30.01 last 121.79) - open/close 5047 (time 1746132625.67 total 40.03 last 128.24) - open/close 6532 (time 1746132635.67 total 50.03 last 148.43) - open/close 7923 (time 1746132645.68 total 60.04 last 139.03) - open/close 9246 (time 1746132655.68 total 70.04 last 132.27) - open/close 10000 (time 1746132661.33 total 75.68 last 133.56) - open/close 11288 (time 1746132671.33 total 85.69 last 128.71) - open/close 12720 (time 1746132681.35 total 95.71 last 142.90) - open/close 14234 (time 1746132691.36 total 105.71 last 151.36) - open/close 15799 (time 1746132701.36 total 115.72 last 156.42) - open/close 17169 (time 1746132711.37 total 125.73 last 136.92) - open/close 18444 (time 1746132721.38 total 135.74 last 127.32) - open/close 19806 (time 1746132731.39 total 145.74 last 136.14) - open/close 21057 (time 1746132741.40 total 155.75 last 124.98) - open/close 22482 (time 1746132751.40 total 165.76 last 142.45) - open/close 23937 (time 1746132761.41 total 175.77 last 145.38) - open/close 25330 (time 1746132771.41 total 185.77 last 139.24) - open/close 26929 (time 1746132781.42 total 195.78 last 159.81) - open/close 28349 (time 1746132791.42 total 205.78 last 141.91) - open/close 29907 (time 1746132801.43 total 215.79 last 155.74) - open/close 31553 (time 1746132811.43 total 225.79 last 164.48) - open/close 33056 (time 1746132821.44 total 235.79 last 150.27) - open/close 34798 (time 1746132831.44 total 245.80 last 174.15) - open/close 36352 (time 1746132841.44 total 255.80 last 155.33) - open/close 37936 (time 1746132851.45 total 265.81 last 158.32) - open/close 39324 (time 1746132861.45 total 275.81 last 138.77) - open/close 40000 (time 1746132866.14 total 280.50 last 144.13) - open/close 41224 (time 1746132876.14 total 290.50 last 122.40) - open/close 42413 (time 1746132886.14 total 300.50 last 118.89) - open/close 43576 (time 1746132896.14 total 310.50 last 116.27) - open/close 44816 (time 1746132906.16 total 320.52 last 123.82) - open/close 46065 (time 1746132916.17 total 330.53 last 124.75) - open/close 47500 (time 1746132926.17 total 340.53 last 143.46) - open/close 49132 (time 1746132936.17 total 350.53 last 163.19) - open/close 50000 (time 1746132941.32 total 355.68 last 168.53) - open/close 51691 (time 1746132951.33 total 365.69 last 169.04) - open/close 53018 (time 1746132961.33 total 375.69 last 132.69) - open/close 54219 (time 1746132971.33 total 385.69 last 120.09) - open/close 55411 (time 1746132981.34 total 395.70 last 119.07) - open/close 56535 (time 1746132991.34 total 405.70 last 112.38) - open/close 57793 (time 1746133001.34 total 415.70 last 125.78) - open/close 59286 (time 1746133011.35 total 425.71 last 149.20) - open/close 60000 (time 1746133016.41 total 430.77 last 141.04) - open/close 61261 (time 1746133026.43 total 440.78 last 125.94) - open/close 62378 (time 1746133036.45 total 450.81 last 111.45) - open/close 63484 (time 1746133046.45 total 460.81 last 110.59) - open/close 64675 (time 1746133056.46 total 470.81 last 119.03) total: 64768 open/close in 471.59 seconds: 137.34 ops/second - /unlink 1963 (time 1746133069.05 total 10.00 last 196.24) - /unlink 3660 (time 1746133079.06 total 20.01 last 169.64) - /unlink 5480 (time 1746133089.06 total 30.01 last 181.98) - /unlink 7602 (time 1746133099.06 total 40.01 last 212.16) - /unlink 9924 (time 1746133109.06 total 50.01 last 232.15) - /unlink 12167 (time 1746133119.07 total 60.01 last 224.24) - /unlink 14207 (time 1746133129.07 total 70.01 last 203.99) - /unlink 16664 (time 1746133139.07 total 80.02 last 245.53) - /unlink 18754 (time 1746133149.08 total 90.02 last 208.94) - /unlink 20000 (time 1746133155.33 total 96.28 last 199.29) - /unlink 22308 (time 1746133165.33 total 106.28 last 230.74) - /unlink 24041 (time 1746133175.33 total 116.28 last 173.29) - /unlink 25774 (time 1746133185.33 total 126.28 last 173.29) - /unlink 27387 (time 1746133195.34 total 136.29 last 161.16) - /unlink 28997 (time 1746133205.34 total 146.29 last 160.99) - /unlink 30000 (time 1746133211.37 total 152.32 last 166.46) - /unlink 31609 (time 1746133221.37 total 162.32 last 160.89) - /unlink 33523 (time 1746133231.37 total 172.32 last 191.34) - /unlink 35194 (time 1746133241.38 total 182.32 last 167.03) - /unlink 37182 (time 1746133251.38 total 192.33 last 198.66) - /unlink 39089 (time 1746133261.38 total 202.33 last 190.65) - /unlink 40000 (time 1746133266.60 total 207.55 last 174.74) - /unlink 42106 (time 1746133276.60 total 217.55 last 210.56) - /unlink 44004 (time 1746133286.60 total 227.55 last 189.78) - /unlink 45979 (time 1746133296.61 total 237.56 last 197.39) - /unlink 47794 (time 1746133306.61 total 247.56 last 181.45) - /unlink 49691 (time 1746133316.61 total 257.56 last 189.66) - /unlink 51761 (time 1746133326.61 total 267.56 last 206.99) - /unlink 53773 (time 1746133336.61 total 277.56 last 201.18) - /unlink 55978 (time 1746133346.62 total 287.57 last 220.42) - /unlink 58378 (time 1746133356.62 total 297.57 last 239.98) - /unlink 60000 (time 1746133362.93 total 303.88 last 256.87) - /unlink 62301 (time 1746133372.94 total 313.89 last 229.95) - /unlink 64061 (time 1746133382.95 total 323.89 last 175.89) total: 64768 /unlink in 328.22 seconds: 197.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1228 (time 1746133415.19 total 10.01 last 122.73) - open/close 2384 (time 1746133425.19 total 20.01 last 115.57) - open/close 3775 (time 1746133435.20 total 30.01 last 139.05) - open/close 5076 (time 1746133445.21 total 40.02 last 129.96) - open/close 6395 (time 1746133455.21 total 50.03 last 131.87) - open/close 7813 (time 1746133465.21 total 60.03 last 141.79) - open/close 8994 (time 1746133475.21 total 70.03 last 118.08) - open/close 10000 (time 1746133482.71 total 77.53 last 134.08) - open/close 11184 (time 1746133492.72 total 87.54 last 118.33) - open/close 12508 (time 1746133502.72 total 97.54 last 132.35) - open/close 13696 (time 1746133512.73 total 107.55 last 118.69) - open/close 14866 (time 1746133522.74 total 117.56 last 116.91) - open/close 16325 (time 1746133532.74 total 127.56 last 145.86) - open/close 17700 (time 1746133542.75 total 137.57 last 137.42) - open/close 19224 (time 1746133552.75 total 147.57 last 152.35) - open/close 20000 (time 1746133556.79 total 151.61 last 192.10) - open/close 21360 (time 1746133566.79 total 161.61 last 135.99) - open/close 22677 (time 1746133576.80 total 171.61 last 131.67) - open/close 23885 (time 1746133586.80 total 181.62 last 120.74) - open/close 25157 (time 1746133596.80 total 191.62 last 127.20) - open/close 26954 (time 1746133606.80 total 201.62 last 179.62) - open/close 28457 (time 1746133616.81 total 211.63 last 150.23) - open/close 30000 (time 1746133626.41 total 221.22 last 160.75) - open/close 31878 (time 1746133636.41 total 231.23 last 187.77) - open/close 33811 (time 1746133646.41 total 241.23 last 193.28) - open/close 35793 (time 1746133656.41 total 251.23 last 198.18) - open/close 37124 (time 1746133666.41 total 261.23 last 133.08) - open/close 38355 (time 1746133676.42 total 271.23 last 123.03) - open/close 39899 (time 1746133686.42 total 281.24 last 154.36) - open/close 41628 (time 1746133696.43 total 291.24 last 172.81) - open/close 43107 (time 1746133706.43 total 301.25 last 147.84) - open/close 44667 (time 1746133716.43 total 311.25 last 155.95) - open/close 46482 (time 1746133726.43 total 321.25 last 181.48) - open/close 48863 (time 1746133736.43 total 331.25 last 238.10) - open/close 50000 (time 1746133742.90 total 337.72 last 175.70) - open/close 51880 (time 1746133752.91 total 347.72 last 187.94) - open/close 53924 (time 1746133762.91 total 357.73 last 204.26) - open/close 55980 (time 1746133772.92 total 367.73 last 205.55) - open/close 57763 (time 1746133782.92 total 377.73 last 178.30) - open/close 60000 (time 1746133792.84 total 387.66 last 225.45) - open/close 62377 (time 1746133802.84 total 397.66 last 237.67) total: 64768 open/close in 407.31 seconds: 159.01 ops/second - /unlink 4321 (time 1746133823.45 total 10.00 last 432.03) - /unlink 8011 (time 1746133833.46 total 20.00 last 368.97) - /unlink 10000 (time 1746133839.45 total 25.99 last 331.99) - /unlink 14711 (time 1746133849.45 total 36.00 last 470.97) - /unlink 19171 (time 1746133859.45 total 46.00 last 445.95) - /unlink 20000 (time 1746133861.55 total 48.10 last 394.73) - /unlink 23176 (time 1746133871.55 total 58.10 last 317.51) - /unlink 26422 (time 1746133881.55 total 68.10 last 324.58) - /unlink 28793 (time 1746133891.56 total 78.10 last 237.02) - /unlink 30000 (time 1746133894.92 total 81.47 last 358.92) - /unlink 33435 (time 1746133904.92 total 91.47 last 343.41) - /unlink 36285 (time 1746133914.93 total 101.47 last 284.88) - /unlink 38742 (time 1746133924.93 total 111.48 last 245.66) - /unlink 40000 (time 1746133930.58 total 117.13 last 222.69) - /unlink 42693 (time 1746133940.58 total 127.13 last 269.27) - /unlink 46170 (time 1746133950.58 total 137.13 last 347.66) - /unlink 50000 (time 1746133957.67 total 144.22 last 540.37) - /unlink 55298 (time 1746133967.67 total 154.22 last 529.72) - /unlink 60000 (time 1746133977.26 total 163.81 last 490.17) - /unlink 64018 (time 1746133987.26 total 173.81 last 401.74) total: 64768 /unlink in 176.21 seconds: 367.56 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1617 (time 1746134017.49 total 10.00 last 161.66) - open/close 3334 (time 1746134027.49 total 20.01 last 171.60) - open/close 5046 (time 1746134037.50 total 30.02 last 171.07) - open/close 6382 (time 1746134047.50 total 40.02 last 133.60) - open/close 8340 (time 1746134057.50 total 50.02 last 195.73) - open/close 10000 (time 1746134066.72 total 59.24 last 180.09) - open/close 11899 (time 1746134076.73 total 69.24 last 189.82) - open/close 13904 (time 1746134086.73 total 79.25 last 200.41) - open/close 15919 (time 1746134096.73 total 89.25 last 201.49) - open/close 18476 (time 1746134106.73 total 99.25 last 255.66) - open/close 20000 (time 1746134116.14 total 108.66 last 161.96) - open/close 21556 (time 1746134126.16 total 118.67 last 155.36) - open/close 22983 (time 1746134136.16 total 128.68 last 142.66) - open/close 24368 (time 1746134146.17 total 138.68 last 138.42) - open/close 25891 (time 1746134156.17 total 148.68 last 152.28) - open/close 27207 (time 1746134166.17 total 158.69 last 131.56) - open/close 28818 (time 1746134176.17 total 168.69 last 161.08) - open/close 30000 (time 1746134183.37 total 175.89 last 164.16) - open/close 31977 (time 1746134193.38 total 185.89 last 197.64) - open/close 34047 (time 1746134203.38 total 195.89 last 206.94) - open/close 35432 (time 1746134213.38 total 205.90 last 138.49) - open/close 36676 (time 1746134223.39 total 215.91 last 124.23) - open/close 37935 (time 1746134233.40 total 225.91 last 125.86) - open/close 39595 (time 1746134243.40 total 235.92 last 165.94) - open/close 40000 (time 1746134245.57 total 238.08 last 186.82) - open/close 42174 (time 1746134255.57 total 248.09 last 217.29) - open/close 43922 (time 1746134265.58 total 258.09 last 174.74) - open/close 45642 (time 1746134275.58 total 268.09 last 171.96) - open/close 47416 (time 1746134285.58 total 278.10 last 177.33) - open/close 49733 (time 1746134295.58 total 288.10 last 231.66) - open/close 52166 (time 1746134305.59 total 298.10 last 243.26) - open/close 54740 (time 1746134315.59 total 308.10 last 257.30) - open/close 56680 (time 1746134325.59 total 318.11 last 193.99) - open/close 59314 (time 1746134335.59 total 328.11 last 263.34) - open/close 60000 (time 1746134338.63 total 331.15 last 225.54) - open/close 62161 (time 1746134348.64 total 341.15 last 216.06) - open/close 63954 (time 1746134358.64 total 351.15 last 179.27) total: 64768 open/close in 356.01 seconds: 181.93 ops/second - /unlink 2384 (time 1746134375.05 total 10.00 last 238.36) - /unlink 5142 (time 1746134385.05 total 20.00 last 275.77) - /unlink 8514 (time 1746134395.05 total 30.00 last 337.17) - /unlink 10000 (time 1746134402.08 total 37.04 last 211.23) - /unlink 12327 (time 1746134412.09 total 47.05 last 232.45) - /unlink 15290 (time 1746134422.09 total 57.05 last 296.28) - /unlink 18956 (time 1746134432.10 total 67.05 last 366.56) - /unlink 20000 (time 1746134435.34 total 70.29 last 322.03) - /unlink 23061 (time 1746134445.34 total 80.29 last 306.05) - /unlink 26801 (time 1746134455.34 total 90.30 last 373.94) - /unlink 29659 (time 1746134465.34 total 100.30 last 285.79) - /unlink 32487 (time 1746134475.34 total 110.30 last 282.70) - /unlink 34532 (time 1746134485.35 total 120.30 last 204.47) - /unlink 36927 (time 1746134495.35 total 130.30 last 239.45) - /unlink 39690 (time 1746134505.35 total 140.31 last 276.26) - /unlink 41887 (time 1746134515.35 total 150.31 last 219.70) - /unlink 44076 (time 1746134525.35 total 160.31 last 218.82) - /unlink 46293 (time 1746134535.35 total 170.31 last 221.69) - /unlink 48824 (time 1746134545.36 total 180.31 last 253.00) - /unlink 50000 (time 1746134549.15 total 184.10 last 310.50) - /unlink 52870 (time 1746134559.15 total 194.10 last 286.96) - /unlink 55550 (time 1746134569.15 total 204.11 last 267.94) - /unlink 58515 (time 1746134579.15 total 214.11 last 296.44) - /unlink 60000 (time 1746134583.95 total 218.91 last 309.39) - /unlink 63499 (time 1746134593.95 total 228.91 last 349.86) total: 64768 /unlink in 233.16 seconds: 277.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1599 (time 1746134625.03 total 10.01 last 159.76) - open/close 2797 (time 1746134635.04 total 20.02 last 119.71) - open/close 4135 (time 1746134645.04 total 30.02 last 133.77) - open/close 5499 (time 1746134655.04 total 40.02 last 136.38) - open/close 6692 (time 1746134665.05 total 50.03 last 119.22) - open/close 8122 (time 1746134675.05 total 60.03 last 142.99) - open/close 9904 (time 1746134685.06 total 70.03 last 178.12) - open/close 11652 (time 1746134695.07 total 80.04 last 174.63) - open/close 12896 (time 1746134705.08 total 90.05 last 124.26) - open/close 14274 (time 1746134715.09 total 100.06 last 137.66) - open/close 15840 (time 1746134725.09 total 110.06 last 156.57) - open/close 17393 (time 1746134735.09 total 120.07 last 155.26) - open/close 18840 (time 1746134745.09 total 130.07 last 144.69) - open/close 20000 (time 1746134754.79 total 139.77 last 119.61) - open/close 21211 (time 1746134764.80 total 149.77 last 120.99) - open/close 22836 (time 1746134774.80 total 159.78 last 162.46) - open/close 24680 (time 1746134784.81 total 169.78 last 184.32) - open/close 26648 (time 1746134794.81 total 179.78 last 196.74) - open/close 29335 (time 1746134804.81 total 189.79 last 268.68) - open/close 32366 (time 1746134814.82 total 199.79 last 302.98) - open/close 33961 (time 1746134824.82 total 209.79 last 159.48) - open/close 35730 (time 1746134834.82 total 219.79 last 176.90) - open/close 37133 (time 1746134844.82 total 229.80 last 140.22) - open/close 38351 (time 1746134854.82 total 239.80 last 121.79) - open/close 39597 (time 1746134864.83 total 249.81 last 124.48) - open/close 40000 (time 1746134868.92 total 253.90 last 98.50) - open/close 41155 (time 1746134878.93 total 263.90 last 115.46) - open/close 42448 (time 1746134888.94 total 273.91 last 129.19) - open/close 43535 (time 1746134898.94 total 283.92 last 108.61) - open/close 44646 (time 1746134908.95 total 293.92 last 111.04) - open/close 45839 (time 1746134918.95 total 303.93 last 119.28) - open/close 47020 (time 1746134928.95 total 313.93 last 118.06) - open/close 48268 (time 1746134938.96 total 323.93 last 124.79) - open/close 49532 (time 1746134948.96 total 333.94 last 126.33) - open/close 50000 (time 1746134952.47 total 337.44 last 133.38) - open/close 51673 (time 1746134962.48 total 347.45 last 167.20) - open/close 53283 (time 1746134972.48 total 357.45 last 161.00) - open/close 55029 (time 1746134982.48 total 367.46 last 174.51) - open/close 56487 (time 1746134992.49 total 377.46 last 145.74) - open/close 58156 (time 1746135002.49 total 387.47 last 166.81) - open/close 59721 (time 1746135012.50 total 397.47 last 156.43) - open/close 60000 (time 1746135014.50 total 399.47 last 139.45) - open/close 62008 (time 1746135024.50 total 409.47 last 200.78) - open/close 63578 (time 1746135034.50 total 419.48 last 156.90) total: 64768 open/close in 428.65 seconds: 151.10 ops/second - /unlink 3121 (time 1746135054.94 total 10.00 last 312.07) - /unlink 5590 (time 1746135064.94 total 20.00 last 246.90) - /unlink 8180 (time 1746135074.94 total 30.00 last 258.96) - /unlink 10000 (time 1746135080.82 total 35.88 last 309.40) - /unlink 13115 (time 1746135090.83 total 45.89 last 311.45) - /unlink 16622 (time 1746135100.83 total 55.89 last 350.61) - /unlink 19030 (time 1746135110.83 total 65.89 last 240.76) - /unlink 20000 (time 1746135115.27 total 70.33 last 218.27) - /unlink 22234 (time 1746135125.27 total 80.34 last 223.38) - /unlink 25042 (time 1746135135.27 total 90.34 last 280.78) - /unlink 27930 (time 1746135145.28 total 100.34 last 288.78) - /unlink 30000 (time 1746135152.32 total 107.38 last 293.92) - /unlink 33513 (time 1746135162.32 total 117.38 last 351.28) - /unlink 37291 (time 1746135172.32 total 127.38 last 377.78) - /unlink 40000 (time 1746135179.29 total 134.35 last 388.47) - /unlink 43369 (time 1746135189.30 total 144.36 last 336.82) - /unlink 46217 (time 1746135199.30 total 154.36 last 284.73) - /unlink 48646 (time 1746135209.30 total 164.36 last 242.85) - /unlink 50000 (time 1746135214.31 total 169.37 last 270.40) - /unlink 52786 (time 1746135224.31 total 179.37 last 278.52) - /unlink 55020 (time 1746135234.31 total 189.37 last 223.37) - /unlink 57978 (time 1746135244.32 total 199.38 last 295.66) - /unlink 60000 (time 1746135251.97 total 207.03 last 264.34) - /unlink 62816 (time 1746135261.97 total 217.03 last 281.60) total: 64768 /unlink in 223.29 seconds: 290.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules via unload_modules_local on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2