== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:26:04 (1746138364) Stopping clients: oleg342-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg342-client.virtnet /mnt/lustre2 (opts:-f) oleg342-server: oleg342-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: detected 4 online CPUs by sysfs oleg342-server: MODOPTS_LIBCFS= oleg342-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg342-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg342-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1789 (time 1746138408.02 total 10.00 last 178.87) - open/close 2973 (time 1746138418.02 total 20.01 last 118.33) - open/close 4234 (time 1746138428.03 total 30.02 last 125.96) - open/close 5380 (time 1746138438.04 total 40.02 last 114.60) - open/close 6823 (time 1746138448.04 total 50.03 last 144.19) - open/close 7922 (time 1746138458.04 total 60.03 last 109.88) - open/close 9222 (time 1746138468.05 total 70.03 last 129.97) - open/close 10000 (time 1746138474.63 total 76.61 last 118.26) - open/close 11149 (time 1746138484.63 total 86.61 last 114.87) - open/close 12288 (time 1746138494.64 total 96.62 last 113.78) - open/close 13444 (time 1746138504.64 total 106.62 last 115.59) - open/close 14658 (time 1746138514.64 total 116.63 last 121.37) - open/close 16240 (time 1746138524.65 total 126.63 last 158.09) - open/close 17928 (time 1746138534.65 total 136.64 last 168.77) - open/close 19517 (time 1746138544.66 total 146.64 last 158.82) - open/close 20000 (time 1746138547.03 total 149.01 last 203.42) - open/close 22269 (time 1746138557.03 total 159.02 last 226.88) - open/close 24627 (time 1746138567.03 total 169.02 last 235.77) - open/close 26763 (time 1746138577.04 total 179.02 last 213.48) - open/close 28520 (time 1746138587.05 total 189.03 last 175.56) - open/close 30000 (time 1746138595.52 total 197.51 last 174.59) - open/close 31646 (time 1746138605.53 total 207.52 last 164.47) - open/close 32864 (time 1746138615.53 total 217.52 last 121.78) - open/close 34056 (time 1746138625.54 total 227.52 last 119.19) - open/close 35084 (time 1746138635.54 total 237.53 last 102.71) - open/close 36240 (time 1746138645.55 total 247.53 last 115.53) - open/close 37354 (time 1746138655.55 total 257.54 last 111.36) - open/close 38598 (time 1746138665.56 total 267.54 last 124.36) - open/close 39851 (time 1746138675.56 total 277.55 last 125.24) - open/close 40880 (time 1746138685.57 total 287.55 last 102.82) - open/close 42008 (time 1746138695.58 total 297.56 last 112.68) - open/close 43092 (time 1746138705.59 total 307.57 last 108.29) - open/close 44074 (time 1746138715.59 total 317.58 last 98.18) - open/close 45276 (time 1746138725.59 total 327.58 last 120.19) - open/close 46385 (time 1746138735.60 total 337.59 last 110.79) - open/close 47586 (time 1746138745.60 total 347.59 last 120.07) - open/close 48685 (time 1746138755.61 total 357.60 last 109.79) - open/close 49718 (time 1746138765.61 total 367.60 last 103.30) - open/close 50000 (time 1746138768.14 total 370.12 last 111.77) - open/close 51146 (time 1746138778.14 total 380.12 last 114.57) - open/close 52277 (time 1746138788.14 total 390.13 last 113.09) - open/close 53398 (time 1746138798.15 total 400.13 last 112.00) - open/close 54308 (time 1746138808.16 total 410.14 last 90.92) - open/close 55351 (time 1746138818.16 total 420.14 last 104.30) - open/close 56354 (time 1746138828.16 total 430.14 last 100.30) - open/close 57450 (time 1746138838.17 total 440.15 last 109.53) - open/close 58601 (time 1746138848.19 total 450.17 last 114.87) - open/close 59808 (time 1746138858.19 total 460.17 last 120.69) - open/close 61079 (time 1746138868.19 total 470.18 last 127.04) - open/close 62148 (time 1746138878.20 total 480.18 last 106.84) - open/close 63366 (time 1746138888.20 total 490.19 last 121.74) - open/close 64640 (time 1746138898.20 total 500.19 last 127.36) total: 64768 open/close in 501.44 seconds: 129.16 ops/second - /unlink 1928 (time 1746138911.22 total 10.00 last 192.79) - /unlink 3840 (time 1746138921.23 total 20.00 last 191.17) - /unlink 5508 (time 1746138931.23 total 30.00 last 166.79) - /unlink 7543 (time 1746138941.23 total 40.00 last 203.45) - /unlink 9280 (time 1746138951.23 total 50.01 last 173.67) - /unlink 10000 (time 1746138954.87 total 53.65 last 197.56) - /unlink 11730 (time 1746138964.88 total 63.65 last 172.98) - /unlink 13596 (time 1746138974.88 total 73.65 last 186.57) - /unlink 15408 (time 1746138984.89 total 83.66 last 181.06) - /unlink 17143 (time 1746138994.89 total 93.66 last 173.47) - /unlink 19022 (time 1746139004.89 total 103.67 last 187.81) - /unlink 20000 (time 1746139010.79 total 109.56 last 165.87) - /unlink 21625 (time 1746139020.79 total 119.56 last 162.49) - /unlink 23300 (time 1746139030.79 total 129.57 last 167.46) - /unlink 25074 (time 1746139040.80 total 139.58 last 177.26) - /unlink 26815 (time 1746139050.80 total 149.58 last 174.07) - /unlink 28573 (time 1746139060.80 total 159.58 last 175.75) - /unlink 30000 (time 1746139069.19 total 167.96 last 170.20) - /unlink 31638 (time 1746139079.19 total 177.96 last 163.79) - /unlink 33284 (time 1746139089.19 total 187.97 last 164.52) - /unlink 34963 (time 1746139099.19 total 197.97 last 167.89) - /unlink 36369 (time 1746139109.20 total 207.97 last 140.58) - /unlink 37875 (time 1746139119.20 total 217.98 last 150.51) - /unlink 39791 (time 1746139129.21 total 227.98 last 191.46) - /unlink 41826 (time 1746139139.21 total 237.99 last 203.41) - /unlink 43604 (time 1746139149.22 total 247.99 last 177.73) - /unlink 45630 (time 1746139159.22 total 258.00 last 202.52) - /unlink 47724 (time 1746139169.22 total 268.00 last 209.40) - /unlink 49857 (time 1746139179.22 total 278.00 last 213.23) - /unlink 51823 (time 1746139189.22 total 288.00 last 196.59) - /unlink 53792 (time 1746139199.23 total 298.00 last 196.85) - /unlink 55696 (time 1746139209.23 total 308.01 last 190.29) - /unlink 57658 (time 1746139219.23 total 318.01 last 196.17) - /unlink 59529 (time 1746139229.24 total 328.02 last 186.99) - /unlink 60000 (time 1746139231.60 total 330.37 last 199.91) - /unlink 62064 (time 1746139241.60 total 340.37 last 206.35) - /unlink 64224 (time 1746139251.60 total 350.38 last 215.93) total: 64768 /unlink in 352.56 seconds: 183.71 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1206 (time 1746139282.52 total 10.01 last 120.50) - open/close 2482 (time 1746139292.52 total 20.01 last 127.59) - open/close 3724 (time 1746139302.52 total 30.01 last 124.17) - open/close 4904 (time 1746139312.53 total 40.02 last 117.91) - open/close 6045 (time 1746139322.53 total 50.02 last 114.04) - open/close 7301 (time 1746139332.54 total 60.03 last 125.54) - open/close 8692 (time 1746139342.54 total 70.03 last 139.10) - open/close 10000 (time 1746139351.61 total 79.10 last 144.18) - open/close 11309 (time 1746139361.61 total 89.10 last 130.88) - open/close 12474 (time 1746139371.61 total 99.11 last 116.48) - open/close 13571 (time 1746139381.61 total 109.11 last 109.68) - open/close 14846 (time 1746139391.62 total 119.11 last 127.45) - open/close 16326 (time 1746139401.62 total 129.11 last 147.98) - open/close 17774 (time 1746139411.62 total 139.11 last 144.79) - open/close 18999 (time 1746139421.63 total 149.13 last 122.34) - open/close 20000 (time 1746139429.84 total 157.33 last 121.96) - open/close 21343 (time 1746139439.84 total 167.33 last 134.30) - open/close 22713 (time 1746139449.85 total 177.34 last 136.94) - open/close 24106 (time 1746139459.85 total 187.34 last 139.23) - open/close 25360 (time 1746139469.86 total 197.35 last 125.33) - open/close 26554 (time 1746139479.86 total 207.36 last 119.31) - open/close 28004 (time 1746139489.87 total 217.36 last 144.92) - open/close 29262 (time 1746139499.88 total 227.37 last 125.72) - open/close 30000 (time 1746139505.47 total 232.96 last 131.90) - open/close 31557 (time 1746139515.49 total 242.98 last 155.44) - open/close 33024 (time 1746139525.49 total 252.99 last 146.62) - open/close 34532 (time 1746139535.50 total 263.00 last 150.64) - open/close 36070 (time 1746139545.51 total 273.00 last 153.71) - open/close 37561 (time 1746139555.51 total 283.01 last 149.04) - open/close 39261 (time 1746139565.52 total 293.02 last 169.85) - open/close 40000 (time 1746139570.15 total 297.64 last 159.72) - open/close 41708 (time 1746139580.15 total 307.64 last 170.79) - open/close 43168 (time 1746139590.15 total 317.65 last 145.94) - open/close 44584 (time 1746139600.16 total 327.65 last 141.56) - open/close 45841 (time 1746139610.17 total 337.66 last 125.60) - open/close 47068 (time 1746139620.17 total 347.66 last 122.68) - open/close 48257 (time 1746139630.17 total 357.67 last 118.83) - open/close 49537 (time 1746139640.18 total 367.67 last 127.96) - open/close 50000 (time 1746139644.81 total 372.30 last 99.89) - open/close 51016 (time 1746139654.81 total 382.31 last 101.58) - open/close 52176 (time 1746139664.82 total 392.31 last 115.93) - open/close 53352 (time 1746139674.83 total 402.32 last 117.53) - open/close 54515 (time 1746139684.83 total 412.32 last 116.29) - open/close 55621 (time 1746139694.83 total 422.33 last 110.51) - open/close 56662 (time 1746139704.84 total 432.33 last 104.06) - open/close 57712 (time 1746139714.84 total 442.33 last 104.96) - open/close 58689 (time 1746139724.85 total 452.34 last 97.60) - open/close 59766 (time 1746139734.86 total 462.35 last 107.63) - open/close 61239 (time 1746139744.86 total 472.36 last 147.20) - open/close 62424 (time 1746139754.87 total 482.36 last 118.43) - open/close 63587 (time 1746139764.87 total 492.37 last 116.28) - open/close 64669 (time 1746139774.88 total 502.37 last 108.16) total: 64768 open/close in 503.21 seconds: 128.71 ops/second - /unlink 1624 (time 1746139787.46 total 10.01 last 162.25) - /unlink 3238 (time 1746139797.47 total 20.02 last 161.25) - /unlink 5240 (time 1746139807.48 total 30.02 last 200.11) - /unlink 7179 (time 1746139817.48 total 40.03 last 193.77) - /unlink 9216 (time 1746139827.49 total 50.04 last 203.57) - /unlink 10000 (time 1746139831.75 total 54.30 last 183.90) - /unlink 12474 (time 1746139841.75 total 64.30 last 247.39) - /unlink 14848 (time 1746139851.76 total 74.30 last 237.37) - /unlink 17018 (time 1746139861.76 total 84.30 last 216.96) - /unlink 19526 (time 1746139871.76 total 94.30 last 250.75) - /unlink 22708 (time 1746139881.76 total 104.31 last 318.18) - /unlink 25038 (time 1746139891.76 total 114.31 last 232.95) - /unlink 26956 (time 1746139901.77 total 124.31 last 191.74) - /unlink 29379 (time 1746139911.77 total 134.31 last 242.28) - /unlink 30000 (time 1746139914.61 total 137.16 last 218.32) - /unlink 32818 (time 1746139924.61 total 147.16 last 281.75) - /unlink 35046 (time 1746139934.61 total 157.16 last 222.74) - /unlink 37118 (time 1746139944.62 total 167.16 last 207.15) - /unlink 38872 (time 1746139954.62 total 177.16 last 175.38) - /unlink 40000 (time 1746139959.77 total 182.31 last 219.12) - /unlink 42064 (time 1746139969.77 total 192.31 last 206.38) - /unlink 43869 (time 1746139979.77 total 202.32 last 180.46) - /unlink 45776 (time 1746139989.77 total 212.32 last 190.70) - /unlink 47754 (time 1746139999.77 total 222.32 last 197.78) - /unlink 49639 (time 1746140009.77 total 232.32 last 188.44) - /unlink 51590 (time 1746140019.78 total 242.32 last 195.09) - /unlink 53317 (time 1746140029.78 total 252.33 last 172.60) - /unlink 55259 (time 1746140039.79 total 262.33 last 194.07) - /unlink 57201 (time 1746140049.79 total 272.33 last 194.18) - /unlink 58959 (time 1746140059.79 total 282.34 last 175.77) - /unlink 60000 (time 1746140066.14 total 288.68 last 164.00) - /unlink 62494 (time 1746140076.14 total 298.69 last 249.33) - /unlink 64383 (time 1746140086.15 total 308.70 last 188.65) total: 64768 /unlink in 311.13 seconds: 208.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1074 (time 1746140118.77 total 10.01 last 107.34) - open/close 2010 (time 1746140128.78 total 20.01 last 93.54) - open/close 3100 (time 1746140138.78 total 30.02 last 108.96) - open/close 4131 (time 1746140148.79 total 40.02 last 103.04) - open/close 5117 (time 1746140158.80 total 50.03 last 98.50) - open/close 6107 (time 1746140168.80 total 60.04 last 98.95) - open/close 7189 (time 1746140178.81 total 70.04 last 108.13) - open/close 8217 (time 1746140188.81 total 80.05 last 102.73) - open/close 9294 (time 1746140198.82 total 90.06 last 107.63) - open/close 10000 (time 1746140205.96 total 97.20 last 98.87) - open/close 11186 (time 1746140215.97 total 107.20 last 118.53) - open/close 12219 (time 1746140225.98 total 117.21 last 103.20) - open/close 13108 (time 1746140235.98 total 127.22 last 88.83) - open/close 14188 (time 1746140245.99 total 137.22 last 107.98) - open/close 15228 (time 1746140255.99 total 147.22 last 104.00) - open/close 16308 (time 1746140265.99 total 157.22 last 108.00) - open/close 17292 (time 1746140275.99 total 167.23 last 98.35) - open/close 18356 (time 1746140286.00 total 177.23 last 106.36) - open/close 19423 (time 1746140296.01 total 187.24 last 106.60) - open/close 20000 (time 1746140301.91 total 193.15 last 97.65) - open/close 21070 (time 1746140311.92 total 203.15 last 106.96) - open/close 22183 (time 1746140321.93 total 213.16 last 111.17) - open/close 23234 (time 1746140331.94 total 223.18 last 104.99) - open/close 24184 (time 1746140341.94 total 233.18 last 94.97) - open/close 25208 (time 1746140351.95 total 243.18 last 102.34) - open/close 26183 (time 1746140361.95 total 253.19 last 97.48) - open/close 27303 (time 1746140371.96 total 263.19 last 111.94) - open/close 28343 (time 1746140381.97 total 273.20 last 103.90) - open/close 29363 (time 1746140391.98 total 283.21 last 101.89) - open/close 30000 (time 1746140397.94 total 289.18 last 106.75) - open/close 31099 (time 1746140407.95 total 299.19 last 109.83) - open/close 32276 (time 1746140417.95 total 309.19 last 117.65) - open/close 33489 (time 1746140427.96 total 319.19 last 121.30) - open/close 34784 (time 1746140437.96 total 329.19 last 129.45) - open/close 35906 (time 1746140447.96 total 339.20 last 112.19) - open/close 36943 (time 1746140457.96 total 349.20 last 103.70) - open/close 38046 (time 1746140467.97 total 359.20 last 110.24) - open/close 39150 (time 1746140477.97 total 369.21 last 110.34) - open/close 40000 (time 1746140485.46 total 376.69 last 113.56) - open/close 41129 (time 1746140495.46 total 386.69 last 112.88) - open/close 42288 (time 1746140505.46 total 396.70 last 115.83) - open/close 43378 (time 1746140515.47 total 406.70 last 108.98) - open/close 44391 (time 1746140525.47 total 416.70 last 101.27) - open/close 45421 (time 1746140535.47 total 426.71 last 102.98) - open/close 46494 (time 1746140545.47 total 436.71 last 107.30) - open/close 47475 (time 1746140555.48 total 446.71 last 98.06) - open/close 48446 (time 1746140565.48 total 456.71 last 97.09) - open/close 49416 (time 1746140575.48 total 466.72 last 96.94) - open/close 50000 (time 1746140580.55 total 471.78 last 115.33) - open/close 51136 (time 1746140590.55 total 481.79 last 113.53) - open/close 52401 (time 1746140600.56 total 491.80 last 126.39) - open/close 53545 (time 1746140610.56 total 501.80 last 114.39) - open/close 54761 (time 1746140620.57 total 511.80 last 121.51) - open/close 55873 (time 1746140630.58 total 521.81 last 111.09) - open/close 56866 (time 1746140640.58 total 531.81 last 99.29) - open/close 58291 (time 1746140650.58 total 541.82 last 142.42) - open/close 59883 (time 1746140660.59 total 551.82 last 159.19) - open/close 61209 (time 1746140670.59 total 561.82 last 132.56) - open/close 62507 (time 1746140680.60 total 571.83 last 129.69) - open/close 63649 (time 1746140690.60 total 581.84 last 114.12) total: 64768 open/close in 590.07 seconds: 109.76 ops/second - /unlink 2369 (time 1746140710.34 total 10.00 last 236.85) - /unlink 5032 (time 1746140720.34 total 20.00 last 266.29) - /unlink 7286 (time 1746140730.35 total 30.01 last 225.27) - /unlink 9318 (time 1746140740.35 total 40.01 last 203.11) - /unlink 10000 (time 1746140744.00 total 43.67 last 186.67) - /unlink 11682 (time 1746140754.01 total 53.67 last 168.10) - /unlink 13317 (time 1746140764.01 total 63.67 last 163.49) - /unlink 15424 (time 1746140774.01 total 73.68 last 210.65) - /unlink 17426 (time 1746140784.02 total 83.68 last 200.13) - /unlink 19084 (time 1746140794.02 total 93.68 last 165.75) - /unlink 20000 (time 1746140798.85 total 98.52 last 189.49) - /unlink 21788 (time 1746140808.86 total 108.52 last 178.74) - /unlink 24007 (time 1746140818.86 total 118.52 last 221.87) - /unlink 26378 (time 1746140828.86 total 128.52 last 237.00) - /unlink 28249 (time 1746140838.87 total 138.53 last 187.00) - /unlink 30000 (time 1746140846.87 total 146.53 last 218.76) - /unlink 32062 (time 1746140856.87 total 156.54 last 206.18) - /unlink 33951 (time 1746140866.87 total 166.54 last 188.87) - /unlink 35689 (time 1746140876.88 total 176.54 last 173.74) - /unlink 37559 (time 1746140886.88 total 186.55 last 186.88) - /unlink 39430 (time 1746140896.89 total 196.55 last 187.02) - /unlink 40000 (time 1746140899.67 total 199.33 last 205.08) - /unlink 42155 (time 1746140909.67 total 209.33 last 215.47) - /unlink 44382 (time 1746140919.67 total 219.33 last 222.65) - /unlink 47027 (time 1746140929.67 total 229.33 last 264.47) - /unlink 49053 (time 1746140939.67 total 239.33 last 202.59) - /unlink 50000 (time 1746140944.10 total 243.76 last 213.89) - /unlink 52572 (time 1746140954.10 total 253.76 last 257.15) - /unlink 54993 (time 1746140964.10 total 263.77 last 242.06) - /unlink 57063 (time 1746140974.11 total 273.77 last 206.98) - /unlink 59260 (time 1746140984.11 total 283.77 last 219.65) - /unlink 60000 (time 1746140988.57 total 288.23 last 166.00) - /unlink 62112 (time 1746140998.57 total 298.23 last 211.19) - /unlink 64054 (time 1746141008.57 total 308.23 last 194.10) total: 64768 /unlink in 311.60 seconds: 207.85 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1133 (time 1746141040.37 total 10.01 last 113.24) - open/close 2272 (time 1746141050.37 total 20.01 last 113.88) - open/close 3300 (time 1746141060.37 total 30.01 last 102.80) - open/close 4283 (time 1746141070.38 total 40.02 last 98.20) - open/close 5170 (time 1746141080.39 total 50.03 last 88.60) - open/close 6073 (time 1746141090.40 total 60.04 last 90.23) - open/close 6977 (time 1746141100.40 total 70.04 last 90.33) - open/close 7910 (time 1746141110.41 total 80.05 last 93.27) - open/close 8848 (time 1746141120.41 total 90.05 last 93.79) - open/close 9769 (time 1746141130.41 total 100.05 last 92.07) - open/close 10000 (time 1746141132.68 total 102.32 last 101.89) - open/close 11005 (time 1746141142.68 total 112.32 last 100.48) - open/close 12167 (time 1746141152.69 total 122.33 last 116.10) - open/close 13240 (time 1746141162.70 total 132.34 last 107.20) - open/close 14381 (time 1746141172.70 total 142.34 last 114.06) - open/close 15517 (time 1746141182.71 total 152.35 last 113.48) - open/close 16546 (time 1746141192.72 total 162.36 last 102.83) - open/close 17609 (time 1746141202.73 total 172.36 last 106.24) - open/close 18665 (time 1746141212.73 total 182.37 last 105.58) - open/close 19650 (time 1746141222.73 total 192.37 last 98.44) - open/close 20000 (time 1746141225.68 total 195.32 last 118.76) - open/close 21142 (time 1746141235.68 total 205.32 last 114.17) - open/close 22342 (time 1746141245.69 total 215.32 last 119.98) - open/close 23462 (time 1746141255.69 total 225.33 last 111.97) - open/close 24585 (time 1746141265.69 total 235.33 last 112.25) - open/close 25490 (time 1746141275.70 total 245.34 last 90.46) - open/close 26499 (time 1746141285.70 total 255.34 last 100.88) - open/close 27573 (time 1746141295.70 total 265.34 last 107.36) - open/close 28652 (time 1746141305.70 total 275.34 last 107.89) - open/close 29853 (time 1746141315.71 total 285.35 last 120.08) - open/close 30886 (time 1746141325.71 total 295.35 last 103.26) - open/close 31981 (time 1746141335.71 total 305.35 last 109.48) - open/close 33222 (time 1746141345.72 total 315.36 last 124.02) - open/close 34256 (time 1746141355.72 total 325.36 last 103.37) - open/close 35335 (time 1746141365.72 total 335.36 last 107.87) - open/close 36587 (time 1746141375.73 total 345.37 last 125.17) - open/close 37690 (time 1746141385.73 total 355.37 last 110.26) - open/close 38801 (time 1746141395.73 total 365.37 last 111.10) - open/close 39985 (time 1746141405.74 total 375.38 last 118.33) - open/close 41238 (time 1746141415.74 total 385.38 last 125.29) - open/close 42303 (time 1746141425.74 total 395.38 last 106.49) - open/close 43729 (time 1746141435.74 total 405.38 last 142.51) - open/close 45042 (time 1746141445.75 total 415.39 last 131.28) - open/close 46183 (time 1746141455.75 total 425.39 last 114.04) - open/close 47315 (time 1746141465.75 total 435.39 last 113.20) - open/close 48650 (time 1746141475.75 total 445.39 last 133.49) - open/close 50000 (time 1746141485.74 total 455.38 last 135.19) - open/close 51378 (time 1746141495.74 total 465.38 last 137.74) - open/close 53049 (time 1746141505.75 total 475.39 last 167.01) - open/close 54728 (time 1746141515.75 total 485.39 last 167.85) - open/close 56122 (time 1746141525.75 total 495.39 last 139.39) - open/close 57656 (time 1746141535.76 total 505.40 last 153.28) - open/close 59329 (time 1746141545.76 total 515.40 last 167.27) - open/close 60000 (time 1746141549.60 total 519.24 last 174.88) - open/close 61453 (time 1746141559.61 total 529.25 last 145.19) - open/close 63058 (time 1746141569.61 total 539.25 last 160.43) - open/close 64492 (time 1746141579.61 total 549.25 last 143.37) total: 64768 open/close in 551.16 seconds: 117.51 ops/second - /unlink 2245 (time 1746141592.88 total 10.01 last 224.35) - /unlink 4909 (time 1746141602.88 total 20.01 last 266.40) - /unlink 7363 (time 1746141612.88 total 30.01 last 245.26) - /unlink 9799 (time 1746141622.89 total 40.02 last 243.48) - /unlink 12212 (time 1746141632.89 total 50.02 last 241.23) - /unlink 15877 (time 1746141642.89 total 60.02 last 366.45) - /unlink 19630 (time 1746141652.89 total 70.02 last 375.28) - /unlink 22435 (time 1746141662.89 total 80.03 last 280.43) - /unlink 25428 (time 1746141672.90 total 90.03 last 299.24) - /unlink 28699 (time 1746141682.90 total 100.03 last 327.08) - /unlink 30000 (time 1746141687.75 total 104.88 last 268.36) - /unlink 32890 (time 1746141697.75 total 114.88 last 288.96) - /unlink 35668 (time 1746141707.75 total 124.88 last 277.80) - /unlink 38002 (time 1746141717.75 total 134.88 last 233.31) - /unlink 40000 (time 1746141726.06 total 143.19 last 240.40) - /unlink 42759 (time 1746141736.06 total 153.19 last 275.85) - /unlink 44974 (time 1746141746.07 total 163.20 last 221.36) - /unlink 47131 (time 1746141756.07 total 173.20 last 215.69) - /unlink 48962 (time 1746141766.07 total 183.20 last 183.09) - /unlink 50000 (time 1746141771.02 total 188.15 last 209.93) - /unlink 52870 (time 1746141781.02 total 198.15 last 286.97) - /unlink 54752 (time 1746141791.02 total 208.15 last 188.15) - /unlink 56818 (time 1746141801.02 total 218.15 last 206.59) - /unlink 58906 (time 1746141811.02 total 228.15 last 208.73) - /unlink 60000 (time 1746141815.77 total 232.90 last 230.36) - /unlink 62805 (time 1746141825.77 total 242.91 last 280.42) total: 64768 /unlink in 250.48 seconds: 258.57 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1366 (time 1746141861.46 total 10.00 last 136.56) - open/close 2669 (time 1746141871.47 total 20.01 last 130.26) - open/close 3949 (time 1746141881.47 total 30.01 last 127.91) - open/close 5245 (time 1746141891.48 total 40.02 last 129.55) - open/close 6677 (time 1746141901.48 total 50.02 last 143.19) - open/close 8355 (time 1746141911.48 total 60.02 last 167.70) - open/close 9811 (time 1746141921.49 total 70.03 last 145.54) - open/close 11258 (time 1746141931.49 total 80.03 last 144.65) - open/close 12675 (time 1746141941.50 total 90.04 last 141.64) - open/close 14066 (time 1746141951.50 total 100.04 last 139.01) - open/close 15387 (time 1746141961.51 total 110.04 last 132.06) - open/close 16805 (time 1746141971.51 total 120.05 last 141.77) - open/close 18173 (time 1746141981.51 total 130.05 last 136.73) - open/close 19788 (time 1746141991.52 total 140.06 last 161.44) - open/close 21230 (time 1746142001.52 total 150.06 last 144.18) - open/close 22369 (time 1746142011.54 total 160.07 last 113.71) - open/close 23609 (time 1746142021.54 total 170.08 last 123.90) - open/close 25264 (time 1746142031.55 total 180.09 last 165.34) - open/close 26851 (time 1746142041.56 total 190.10 last 158.65) - open/close 28231 (time 1746142051.56 total 200.10 last 137.94) - open/close 29743 (time 1746142061.56 total 210.10 last 151.17) - open/close 31196 (time 1746142071.56 total 220.10 last 145.27) - open/close 32756 (time 1746142081.57 total 230.11 last 155.92) - open/close 34666 (time 1746142091.58 total 240.12 last 190.84) - open/close 35996 (time 1746142101.58 total 250.12 last 133.00) - open/close 37329 (time 1746142111.58 total 260.12 last 133.26) - open/close 39061 (time 1746142121.58 total 270.12 last 173.15) - open/close 40000 (time 1746142128.01 total 276.55 last 146.01) - open/close 41688 (time 1746142138.02 total 286.56 last 168.74) - open/close 43703 (time 1746142148.02 total 296.56 last 201.49) - open/close 45353 (time 1746142158.03 total 306.56 last 164.88) - open/close 46896 (time 1746142168.03 total 316.57 last 154.19) - open/close 48335 (time 1746142178.04 total 326.58 last 143.82) - open/close 49666 (time 1746142188.04 total 336.58 last 133.02) - open/close 50000 (time 1746142190.39 total 338.93 last 142.45) - open/close 51607 (time 1746142200.39 total 348.93 last 160.68) - open/close 53114 (time 1746142210.39 total 358.93 last 150.70) - open/close 54600 (time 1746142220.39 total 368.93 last 148.54) - open/close 56505 (time 1746142230.40 total 378.94 last 190.40) - open/close 58431 (time 1746142240.40 total 388.94 last 192.59) - open/close 60000 (time 1746142249.86 total 398.40 last 165.85) - open/close 61317 (time 1746142259.87 total 408.41 last 131.58) - open/close 63115 (time 1746142269.87 total 418.41 last 179.79) - open/close 64691 (time 1746142279.87 total 428.41 last 157.57) total: 64768 open/close in 428.96 seconds: 150.99 ops/second - /unlink 2369 (time 1746142292.14 total 10.00 last 236.89) - /unlink 5208 (time 1746142302.14 total 20.00 last 283.81) - /unlink 7772 (time 1746142312.14 total 30.01 last 256.35) - /unlink 10000 (time 1746142320.68 total 38.54 last 260.92) - /unlink 12532 (time 1746142330.68 total 48.55 last 253.15) - /unlink 15446 (time 1746142340.68 total 58.55 last 291.36) - /unlink 18487 (time 1746142350.68 total 68.55 last 304.09) - /unlink 20000 (time 1746142354.90 total 72.77 last 358.74) - /unlink 22370 (time 1746142364.90 total 82.77 last 236.96) - /unlink 24564 (time 1746142374.90 total 92.77 last 219.39) - /unlink 26701 (time 1746142384.91 total 102.77 last 213.64) - /unlink 29002 (time 1746142394.91 total 112.77 last 230.05) - /unlink 30000 (time 1746142399.09 total 116.95 last 238.86) - /unlink 32261 (time 1746142409.09 total 126.95 last 226.07) - /unlink 34718 (time 1746142419.09 total 136.95 last 245.66) - /unlink 37054 (time 1746142429.09 total 146.96 last 233.53) - /unlink 39444 (time 1746142439.09 total 156.96 last 238.98) - /unlink 40000 (time 1746142441.27 total 159.13 last 255.77) - /unlink 42334 (time 1746142451.27 total 169.13 last 233.36) - /unlink 44579 (time 1746142461.27 total 179.14 last 224.45) - /unlink 46880 (time 1746142471.27 total 189.14 last 230.10) - /unlink 49319 (time 1746142481.27 total 199.14 last 243.87) - /unlink 50000 (time 1746142484.00 total 201.86 last 249.71) - /unlink 52362 (time 1746142494.00 total 211.87 last 236.17) - /unlink 54779 (time 1746142504.00 total 221.87 last 241.67) - /unlink 57453 (time 1746142514.00 total 231.87 last 267.37) - /unlink 59985 (time 1746142524.00 total 241.87 last 253.16) - /unlink 62309 (time 1746142534.01 total 251.87 last 232.34) - /unlink 64538 (time 1746142544.01 total 261.87 last 222.84) total: 64768 /unlink in 262.96 seconds: 246.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server stop mds service on oleg342-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server unloading modules via unload_modules_local on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2