== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 00:01:39 (1746072099) Stopping clients: oleg323-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg323-client.virtnet /mnt/lustre2 (opts:-f) oleg323-server: oleg323-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: MODOPTS_LIBCFS= oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg323-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg323-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 336 65472 1% /mnt/lustre[OST:0] filesystem_summary: 66001 529 65472 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1036 (time 1746072171.94 total 10.00 last 103.60) - open/close 2148 (time 1746072181.95 total 20.01 last 111.13) - open/close 3342 (time 1746072191.95 total 30.01 last 119.39) - open/close 4561 (time 1746072201.95 total 40.01 last 121.87) - open/close 5759 (time 1746072211.97 total 50.03 last 119.60) - open/close 7090 (time 1746072221.97 total 60.03 last 133.06) - open/close 8489 (time 1746072231.97 total 70.03 last 139.85) - open/close 9954 (time 1746072241.98 total 80.04 last 146.41) - open/close 11286 (time 1746072251.98 total 90.04 last 133.15) - open/close 12440 (time 1746072261.99 total 100.05 last 115.34) - open/close 13562 (time 1746072271.99 total 110.05 last 112.12) - open/close 14905 (time 1746072282.00 total 120.06 last 134.23) - open/close 16217 (time 1746072292.01 total 130.07 last 131.05) - open/close 17453 (time 1746072302.02 total 140.08 last 123.51) - open/close 18744 (time 1746072312.04 total 150.10 last 128.77) - open/close 19949 (time 1746072322.05 total 160.11 last 120.38) - open/close 21269 (time 1746072332.06 total 170.12 last 131.98) - open/close 22712 (time 1746072342.06 total 180.12 last 144.29) - open/close 23919 (time 1746072352.06 total 190.12 last 120.63) - open/close 25411 (time 1746072362.06 total 200.12 last 149.19) - open/close 26869 (time 1746072372.07 total 210.13 last 145.75) - open/close 28201 (time 1746072382.07 total 220.14 last 133.09) - open/close 29381 (time 1746072392.09 total 230.15 last 117.79) - open/close 30000 (time 1746072397.57 total 235.63 last 113.07) - open/close 31306 (time 1746072407.57 total 245.63 last 130.56) - open/close 32640 (time 1746072417.57 total 255.63 last 133.38) - open/close 34140 (time 1746072427.57 total 265.64 last 149.94) - open/close 35500 (time 1746072437.59 total 275.65 last 135.80) - open/close 36887 (time 1746072447.59 total 285.65 last 138.64) - open/close 38142 (time 1746072457.60 total 295.66 last 125.38) - open/close 39589 (time 1746072467.61 total 305.67 last 144.64) - open/close 40000 (time 1746072470.81 total 308.87 last 128.42) - open/close 41179 (time 1746072480.81 total 318.87 last 117.84) - open/close 42553 (time 1746072490.82 total 328.88 last 137.35) - open/close 43934 (time 1746072500.82 total 338.88 last 138.04) - open/close 45207 (time 1746072510.83 total 348.89 last 127.23) - open/close 46285 (time 1746072520.83 total 358.89 last 107.80) - open/close 47382 (time 1746072530.83 total 368.89 last 109.66) - open/close 48482 (time 1746072540.83 total 378.90 last 109.96) - open/close 49553 (time 1746072550.85 total 388.91 last 106.99) - open/close 50000 (time 1746072554.86 total 392.92 last 111.24) - open/close 51140 (time 1746072564.87 total 402.93 last 113.88) - open/close 52727 (time 1746072574.88 total 412.94 last 158.68) - open/close 54169 (time 1746072584.88 total 422.94 last 144.20) - open/close 55589 (time 1746072594.88 total 432.94 last 141.99) - open/close 57071 (time 1746072604.88 total 442.94 last 148.13) - open/close 58446 (time 1746072614.88 total 452.95 last 137.45) - open/close 60000 (time 1746072624.61 total 462.68 last 159.71) - open/close 61268 (time 1746072634.61 total 472.68 last 126.79) - open/close 63052 (time 1746072644.62 total 482.68 last 178.40) - open/close 64650 (time 1746072654.62 total 492.68 last 159.74) total: 64768 open/close in 493.43 seconds: 131.26 ops/second - /unlink 2820 (time 1746072667.01 total 10.00 last 281.99) - /unlink 5281 (time 1746072677.01 total 20.00 last 246.04) - /unlink 7185 (time 1746072687.01 total 30.01 last 190.35) - /unlink 9392 (time 1746072697.01 total 40.01 last 220.68) - /unlink 10000 (time 1746072699.69 total 42.69 last 226.84) - /unlink 12240 (time 1746072709.70 total 52.69 last 223.85) - /unlink 14532 (time 1746072719.71 total 62.70 last 229.01) - /unlink 16829 (time 1746072729.71 total 72.70 last 229.66) - /unlink 19252 (time 1746072739.73 total 82.72 last 241.96) - /unlink 20000 (time 1746072742.96 total 85.95 last 231.12) - /unlink 22197 (time 1746072752.96 total 95.96 last 219.68) - /unlink 24562 (time 1746072762.97 total 105.96 last 236.44) - /unlink 26999 (time 1746072772.97 total 115.96 last 243.63) - /unlink 29595 (time 1746072782.97 total 125.97 last 259.47) - /unlink 30000 (time 1746072785.33 total 128.32 last 171.73) - /unlink 32250 (time 1746072795.33 total 138.32 last 224.98) - /unlink 34347 (time 1746072805.34 total 148.33 last 209.63) - /unlink 36508 (time 1746072815.34 total 158.33 last 216.09) - /unlink 38717 (time 1746072825.34 total 168.33 last 220.83) - /unlink 40000 (time 1746072831.50 total 174.49 last 208.36) - /unlink 42110 (time 1746072841.50 total 184.49 last 210.95) - /unlink 44204 (time 1746072851.50 total 194.49 last 209.39) - /unlink 46215 (time 1746072861.50 total 204.49 last 201.05) - /unlink 48446 (time 1746072871.50 total 214.50 last 223.05) - /unlink 50000 (time 1746072878.13 total 221.12 last 234.50) - /unlink 52026 (time 1746072888.13 total 231.12 last 202.60) - /unlink 53998 (time 1746072898.13 total 241.13 last 197.16) - /unlink 56035 (time 1746072908.14 total 251.13 last 203.56) - /unlink 57659 (time 1746072918.14 total 261.13 last 162.39) - /unlink 59549 (time 1746072928.15 total 271.14 last 188.90) - /unlink 60000 (time 1746072930.62 total 273.61 last 182.38) - /unlink 61854 (time 1746072940.62 total 283.61 last 185.39) - /unlink 63669 (time 1746072950.62 total 293.61 last 181.45) total: 64768 /unlink in 299.58 seconds: 216.20 ops/second Waiting for MDT destroys to complete - open/close 1209 (time 1746072977.59 total 10.00 last 120.85) - open/close 2334 (time 1746072987.59 total 20.01 last 112.48) - open/close 3416 (time 1746072997.59 total 30.01 last 108.19) - open/close 4690 (time 1746073007.59 total 40.01 last 127.40) - open/close 6074 (time 1746073017.59 total 50.01 last 138.40) - open/close 7188 (time 1746073027.60 total 60.02 last 111.30) - open/close 8366 (time 1746073037.60 total 70.02 last 117.78) - open/close 9744 (time 1746073047.61 total 80.02 last 137.73) - open/close 10000 (time 1746073049.79 total 82.21 last 117.25) - open/close 11234 (time 1746073059.79 total 92.21 last 123.39) - open/close 12469 (time 1746073069.80 total 102.21 last 123.46) - open/close 14006 (time 1746073079.81 total 112.22 last 153.56) - open/close 15204 (time 1746073089.81 total 122.22 last 119.79) - open/close 17055 (time 1746073099.81 total 132.22 last 185.08) - open/close 18461 (time 1746073109.81 total 142.23 last 140.54) - open/close 19711 (time 1746073119.82 total 152.23 last 124.94) - open/close 20000 (time 1746073122.05 total 154.47 last 129.28) - open/close 21997 (time 1746073132.06 total 164.47 last 199.63) - open/close 23317 (time 1746073142.06 total 174.47 last 131.99) - open/close 24711 (time 1746073152.06 total 184.48 last 139.34) - open/close 26060 (time 1746073162.07 total 194.48 last 134.83) - open/close 27356 (time 1746073172.07 total 204.48 last 129.56) - open/close 28548 (time 1746073182.08 total 214.49 last 119.08) - open/close 29761 (time 1746073192.08 total 224.50 last 121.24) - open/close 31114 (time 1746073202.09 total 234.51 last 135.19) - open/close 32157 (time 1746073212.10 total 244.51 last 104.24) - open/close 33342 (time 1746073222.10 total 254.51 last 118.49) - open/close 34705 (time 1746073232.11 total 264.52 last 136.14) - open/close 35829 (time 1746073242.12 total 274.54 last 112.28) - open/close 37058 (time 1746073252.12 total 284.54 last 122.88) - open/close 38513 (time 1746073262.13 total 294.54 last 145.44) - open/close 39638 (time 1746073272.13 total 304.55 last 112.45) - open/close 40000 (time 1746073275.26 total 307.68 last 115.53) - open/close 41328 (time 1746073285.27 total 317.68 last 132.77) - open/close 42536 (time 1746073295.27 total 327.69 last 120.71) - open/close 43571 (time 1746073305.29 total 337.70 last 103.38) - open/close 44724 (time 1746073315.29 total 347.70 last 115.29) - open/close 45998 (time 1746073325.29 total 357.70 last 127.40) - open/close 47107 (time 1746073335.29 total 367.71 last 110.85) - open/close 48172 (time 1746073345.29 total 377.71 last 106.49) - open/close 49299 (time 1746073355.29 total 387.71 last 112.68) - open/close 50000 (time 1746073360.95 total 393.37 last 123.90) - open/close 51194 (time 1746073370.96 total 403.37 last 119.31) - open/close 52231 (time 1746073380.96 total 413.38 last 103.65) - open/close 53241 (time 1746073390.97 total 423.39 last 100.91) - open/close 54279 (time 1746073400.98 total 433.39 last 103.77) - open/close 55474 (time 1746073410.98 total 443.39 last 119.50) - open/close 56752 (time 1746073420.98 total 453.39 last 127.80) - open/close 58161 (time 1746073430.98 total 463.39 last 140.84) - open/close 59420 (time 1746073440.99 total 473.40 last 125.80) - open/close 60000 (time 1746073445.91 total 478.32 last 117.91) - open/close 61273 (time 1746073455.91 total 488.32 last 127.29) - open/close 62668 (time 1746073465.91 total 498.32 last 139.49) - open/close 63852 (time 1746073475.91 total 508.32 last 118.39) total: 64768 open/close in 516.09 seconds: 125.50 ops/second - /unlink 2376 (time 1746073495.74 total 10.00 last 237.54) - /unlink 4376 (time 1746073505.75 total 20.01 last 199.84) - /unlink 6293 (time 1746073515.75 total 30.01 last 191.64) - /unlink 8438 (time 1746073525.75 total 40.02 last 214.44) - /unlink 10000 (time 1746073533.03 total 47.30 last 214.51) - /unlink 12076 (time 1746073543.04 total 57.30 last 207.50) - /unlink 14593 (time 1746073553.04 total 67.31 last 251.62) - /unlink 17359 (time 1746073563.05 total 77.31 last 276.41) - /unlink 19670 (time 1746073573.05 total 87.32 last 231.06) - /unlink 21847 (time 1746073583.06 total 97.32 last 217.52) - /unlink 24051 (time 1746073593.06 total 107.32 last 220.40) - /unlink 26667 (time 1746073603.06 total 117.33 last 261.50) - /unlink 28869 (time 1746073613.06 total 127.33 last 220.19) - /unlink 30000 (time 1746073617.77 total 132.04 last 240.14) - /unlink 32471 (time 1746073627.78 total 142.04 last 247.02) - /unlink 34888 (time 1746073637.79 total 152.05 last 241.43) - /unlink 37178 (time 1746073647.79 total 162.05 last 228.97) - /unlink 39559 (time 1746073657.79 total 172.06 last 238.02) - /unlink 42220 (time 1746073667.79 total 182.06 last 266.06) - /unlink 44699 (time 1746073677.80 total 192.06 last 247.82) - /unlink 46978 (time 1746073687.80 total 202.06 last 227.85) - /unlink 49334 (time 1746073697.80 total 212.07 last 235.49) - /unlink 50000 (time 1746073700.49 total 214.76 last 247.73) - /unlink 52302 (time 1746073710.49 total 224.76 last 230.14) - /unlink 54552 (time 1746073720.50 total 234.76 last 224.94) - /unlink 56695 (time 1746073730.50 total 244.77 last 214.24) - /unlink 58807 (time 1746073740.50 total 254.77 last 211.18) - /unlink 60000 (time 1746073745.77 total 260.03 last 226.60) - /unlink 62290 (time 1746073755.77 total 270.03 last 228.94) - /unlink 64656 (time 1746073765.78 total 280.04 last 236.43) total: 64768 /unlink in 280.58 seconds: 230.84 ops/second Waiting for MDT destroys to complete - open/close 1681 (time 1746073788.89 total 10.00 last 168.10) - open/close 3168 (time 1746073798.89 total 20.01 last 148.60) - open/close 4933 (time 1746073808.90 total 30.02 last 176.34) - open/close 6321 (time 1746073818.91 total 40.02 last 138.71) - open/close 7846 (time 1746073828.91 total 50.03 last 152.44) - open/close 9520 (time 1746073838.92 total 60.03 last 167.33) - open/close 10000 (time 1746073842.14 total 63.25 last 149.05) - open/close 11508 (time 1746073852.14 total 73.25 last 150.79) - open/close 13091 (time 1746073862.14 total 83.26 last 158.24) - open/close 14605 (time 1746073872.15 total 93.27 last 151.22) - open/close 16058 (time 1746073882.16 total 103.27 last 145.26) - open/close 17541 (time 1746073892.16 total 113.28 last 148.22) - open/close 19409 (time 1746073902.16 total 123.28 last 186.77) - open/close 20000 (time 1746073905.70 total 126.81 last 167.07) - open/close 21484 (time 1746073915.71 total 136.82 last 148.35) - open/close 22915 (time 1746073925.71 total 146.82 last 143.04) - open/close 24229 (time 1746073935.71 total 156.83 last 131.35) - open/close 25489 (time 1746073945.72 total 166.83 last 125.94) - open/close 26899 (time 1746073955.72 total 176.84 last 140.94) - open/close 28322 (time 1746073965.72 total 186.84 last 142.27) - open/close 29999 (time 1746073975.72 total 196.84 last 167.70) - open/close 31406 (time 1746073985.73 total 206.84 last 140.66) - open/close 32772 (time 1746073995.73 total 216.85 last 136.52) - open/close 34210 (time 1746074005.74 total 226.85 last 143.71) - open/close 35636 (time 1746074015.74 total 236.86 last 142.55) - open/close 36985 (time 1746074025.75 total 246.86 last 134.85) - open/close 38564 (time 1746074035.75 total 256.86 last 157.89) - open/close 39981 (time 1746074045.75 total 266.87 last 141.61) - open/close 41551 (time 1746074055.76 total 276.87 last 156.91) - open/close 42888 (time 1746074065.76 total 286.88 last 133.63) - open/close 44123 (time 1746074075.77 total 296.89 last 123.41) - open/close 45524 (time 1746074085.77 total 306.89 last 140.08) - open/close 46773 (time 1746074095.78 total 316.89 last 124.85) - open/close 48060 (time 1746074105.78 total 326.89 last 128.70) - open/close 49369 (time 1746074115.79 total 336.90 last 130.71) - open/close 50000 (time 1746074120.72 total 341.83 last 128.02) - open/close 51395 (time 1746074130.72 total 351.84 last 139.44) - open/close 52831 (time 1746074140.72 total 361.84 last 143.59) - open/close 54118 (time 1746074150.73 total 371.84 last 128.62) - open/close 55501 (time 1746074160.74 total 381.85 last 138.23) - open/close 56866 (time 1746074170.74 total 391.85 last 136.50) - open/close 58156 (time 1746074180.74 total 401.86 last 128.89) - open/close 59527 (time 1746074190.75 total 411.86 last 137.06) - open/close 60000 (time 1746074194.31 total 415.42 last 132.74) - open/close 61322 (time 1746074204.32 total 425.43 last 132.12) - open/close 62600 (time 1746074214.32 total 435.43 last 127.74) - open/close 63838 (time 1746074224.33 total 445.44 last 123.74) total: 64768 open/close in 452.14 seconds: 143.25 ops/second - /unlink 2049 (time 1746074242.81 total 10.01 last 204.74) - /unlink 4037 (time 1746074252.81 total 20.01 last 198.79) - /unlink 6227 (time 1746074262.82 total 30.01 last 218.94) - /unlink 8550 (time 1746074272.82 total 40.01 last 232.24) - /unlink 10000 (time 1746074281.54 total 48.74 last 166.22) - /unlink 11962 (time 1746074291.54 total 58.74 last 196.18) - /unlink 13807 (time 1746074301.55 total 68.74 last 184.42) - /unlink 15553 (time 1746074311.55 total 78.74 last 174.59) - /unlink 17363 (time 1746074321.55 total 88.75 last 180.94) - /unlink 19154 (time 1746074331.55 total 98.75 last 179.09) - /unlink 20000 (time 1746074335.65 total 102.85 last 206.33) - /unlink 22173 (time 1746074345.66 total 112.85 last 217.20) - /unlink 24569 (time 1746074355.66 total 122.86 last 239.52) - /unlink 26750 (time 1746074365.66 total 132.86 last 218.05) - /unlink 28902 (time 1746074375.67 total 142.86 last 215.15) - /unlink 30000 (time 1746074381.10 total 148.29 last 202.13) - /unlink 32183 (time 1746074391.10 total 158.30 last 218.22) - /unlink 34334 (time 1746074401.10 total 168.30 last 215.05) - /unlink 36484 (time 1746074411.11 total 178.30 last 214.97) - /unlink 38677 (time 1746074421.11 total 188.30 last 219.26) - /unlink 40000 (time 1746074428.07 total 195.27 last 189.96) - /unlink 42007 (time 1746074438.07 total 205.27 last 200.69) - /unlink 43946 (time 1746074448.07 total 215.27 last 193.84) - /unlink 46255 (time 1746074458.08 total 225.28 last 230.78) - /unlink 48360 (time 1746074468.08 total 235.28 last 210.44) - /unlink 50000 (time 1746074475.25 total 242.45 last 228.73) - /unlink 52489 (time 1746074485.26 total 252.45 last 248.85) - /unlink 54956 (time 1746074495.26 total 262.45 last 246.63) - /unlink 57036 (time 1746074505.26 total 272.45 last 207.97) - /unlink 59540 (time 1746074515.26 total 282.46 last 250.35) - /unlink 61716 (time 1746074525.27 total 292.46 last 217.49) - /unlink 64160 (time 1746074535.27 total 302.46 last 244.34) total: 64768 /unlink in 304.41 seconds: 212.76 ops/second Waiting for MDT destroys to complete - open/close 1306 (time 1746074558.87 total 10.00 last 130.54) - open/close 2731 (time 1746074568.87 total 20.01 last 142.45) - open/close 4130 (time 1746074578.88 total 30.01 last 139.82) - open/close 5458 (time 1746074588.88 total 40.02 last 132.76) - open/close 6800 (time 1746074598.89 total 50.02 last 134.11) - open/close 8191 (time 1746074608.90 total 60.04 last 138.93) - open/close 9580 (time 1746074618.90 total 70.04 last 138.89) - open/close 10000 (time 1746074622.25 total 73.38 last 125.54) - open/close 11413 (time 1746074632.25 total 83.38 last 141.29) - open/close 12712 (time 1746074642.26 total 93.39 last 129.84) - open/close 13927 (time 1746074652.26 total 103.40 last 121.42) - open/close 15224 (time 1746074662.26 total 113.40 last 129.67) - open/close 16599 (time 1746074672.27 total 123.40 last 137.42) - open/close 17925 (time 1746074682.27 total 133.40 last 132.59) - open/close 19362 (time 1746074692.27 total 143.41 last 143.68) - open/close 20000 (time 1746074696.43 total 147.56 last 153.59) - open/close 21306 (time 1746074706.43 total 157.56 last 130.55) - open/close 22538 (time 1746074716.43 total 167.57 last 123.17) - open/close 23975 (time 1746074726.43 total 177.57 last 143.69) - open/close 25291 (time 1746074736.44 total 187.57 last 131.53) - open/close 26627 (time 1746074746.44 total 197.57 last 133.58) - open/close 28073 (time 1746074756.44 total 207.57 last 144.58) - open/close 29356 (time 1746074766.44 total 217.58 last 128.28) - open/close 30000 (time 1746074770.57 total 221.71 last 155.87) - open/close 31578 (time 1746074780.58 total 231.72 last 157.64) - open/close 32798 (time 1746074790.59 total 241.72 last 121.93) - open/close 34144 (time 1746074800.60 total 251.73 last 134.45) - open/close 35565 (time 1746074810.60 total 261.74 last 142.07) - open/close 36983 (time 1746074820.61 total 271.74 last 141.73) - open/close 38614 (time 1746074830.61 total 281.75 last 163.02) - open/close 40000 (time 1746074840.56 total 291.70 last 139.28) - open/close 41544 (time 1746074850.58 total 301.71 last 154.22) - open/close 43193 (time 1746074860.58 total 311.71 last 164.89) - open/close 44907 (time 1746074870.59 total 321.72 last 171.26) - open/close 46379 (time 1746074880.59 total 331.72 last 147.17) - open/close 47844 (time 1746074890.59 total 341.72 last 146.49) - open/close 49135 (time 1746074900.60 total 351.73 last 128.98) - open/close 50000 (time 1746074908.84 total 359.97 last 104.95) - open/close 51102 (time 1746074918.84 total 369.98 last 110.17) - open/close 52428 (time 1746074928.84 total 379.98 last 132.60) - open/close 53736 (time 1746074938.85 total 389.98 last 130.76) - open/close 55076 (time 1746074948.85 total 399.98 last 133.96) - open/close 56352 (time 1746074958.85 total 409.98 last 127.56) - open/close 57627 (time 1746074968.85 total 419.99 last 127.46) - open/close 59138 (time 1746074978.85 total 429.99 last 151.10) - open/close 60000 (time 1746074985.17 total 436.30 last 136.54) - open/close 61362 (time 1746074995.17 total 446.31 last 136.13) - open/close 62871 (time 1746075005.18 total 456.31 last 150.85) - open/close 64156 (time 1746075015.18 total 466.31 last 128.50) total: 64768 open/close in 470.80 seconds: 137.57 ops/second - /unlink 2448 (time 1746075031.46 total 10.00 last 244.76) - /unlink 4801 (time 1746075041.47 total 20.01 last 235.21) - /unlink 6845 (time 1746075051.48 total 30.02 last 204.16) - /unlink 9196 (time 1746075061.48 total 40.02 last 235.05) - /unlink 10000 (time 1746075065.00 total 43.54 last 228.33) - /unlink 12079 (time 1746075075.01 total 53.55 last 207.72) - /unlink 14264 (time 1746075085.02 total 63.55 last 218.42) - /unlink 16457 (time 1746075095.02 total 73.56 last 219.24) - /unlink 18546 (time 1746075105.02 total 83.56 last 208.90) - /unlink 20000 (time 1746075111.73 total 90.27 last 216.55) - /unlink 22327 (time 1746075121.74 total 100.27 last 232.64) - /unlink 24559 (time 1746075131.74 total 110.28 last 223.10) - /unlink 26614 (time 1746075141.75 total 120.29 last 205.32) - /unlink 28927 (time 1746075151.75 total 130.29 last 231.25) - /unlink 30000 (time 1746075156.50 total 135.04 last 225.82) - /unlink 32092 (time 1746075166.54 total 145.08 last 208.42) - /unlink 34351 (time 1746075176.54 total 155.08 last 225.81) - /unlink 36795 (time 1746075186.55 total 165.08 last 244.37) - /unlink 38905 (time 1746075196.55 total 175.09 last 210.87) - /unlink 40000 (time 1746075201.86 total 180.40 last 206.35) - /unlink 42224 (time 1746075211.86 total 190.40 last 222.37) - /unlink 44475 (time 1746075221.86 total 200.40 last 225.07) - /unlink 46715 (time 1746075231.86 total 210.40 last 223.98) - /unlink 49055 (time 1746075241.88 total 220.41 last 233.65) - /unlink 50000 (time 1746075245.97 total 224.51 last 230.74) - /unlink 52092 (time 1746075255.97 total 234.51 last 209.19) - /unlink 54174 (time 1746075265.98 total 244.51 last 208.09) - /unlink 56472 (time 1746075275.98 total 254.51 last 229.80) - /unlink 58628 (time 1746075285.98 total 264.52 last 215.55) - /unlink 60000 (time 1746075292.33 total 270.86 last 216.22) - /unlink 62180 (time 1746075302.33 total 280.86 last 217.97) - /unlink 64312 (time 1746075312.33 total 290.87 last 213.08) total: 64768 /unlink in 292.72 seconds: 221.26 ops/second Waiting for MDT destroys to complete - open/close 1293 (time 1746075336.78 total 10.01 last 129.23) - open/close 2632 (time 1746075346.78 total 20.01 last 133.85) - open/close 3838 (time 1746075356.78 total 30.01 last 120.55) - open/close 5106 (time 1746075366.79 total 40.02 last 126.75) - open/close 6521 (time 1746075376.79 total 50.02 last 141.49) - open/close 7796 (time 1746075386.79 total 60.02 last 127.44) - open/close 9146 (time 1746075396.79 total 70.02 last 134.98) - open/close 10000 (time 1746075403.46 total 76.69 last 128.14) - open/close 11328 (time 1746075413.46 total 86.69 last 132.77) - open/close 12634 (time 1746075423.46 total 96.69 last 130.58) - open/close 13874 (time 1746075433.47 total 106.70 last 123.89) - open/close 15348 (time 1746075443.48 total 116.70 last 147.35) - open/close 16681 (time 1746075453.48 total 126.70 last 133.28) - open/close 18104 (time 1746075463.48 total 136.71 last 142.26) - open/close 19550 (time 1746075473.48 total 146.71 last 144.58) - open/close 20000 (time 1746075476.68 total 149.91 last 140.70) - open/close 21303 (time 1746075486.68 total 159.91 last 130.23) - open/close 22672 (time 1746075496.69 total 169.91 last 136.88) - open/close 23993 (time 1746075506.69 total 179.92 last 132.09) - open/close 25341 (time 1746075516.69 total 189.92 last 134.74) - open/close 26641 (time 1746075526.69 total 199.92 last 129.98) - open/close 28037 (time 1746075536.70 total 209.93 last 139.53) - open/close 29295 (time 1746075546.70 total 219.93 last 125.74) - open/close 30000 (time 1746075551.98 total 225.21 last 133.49) - open/close 31273 (time 1746075561.99 total 235.22 last 127.18) - open/close 32665 (time 1746075571.99 total 245.22 last 139.19) - open/close 33847 (time 1746075582.00 total 255.23 last 118.16) - open/close 35316 (time 1746075592.00 total 265.23 last 146.84) - open/close 36803 (time 1746075602.00 total 275.23 last 148.65) - open/close 38192 (time 1746075612.01 total 285.24 last 138.77) - open/close 39610 (time 1746075622.01 total 295.24 last 141.79) - open/close 40000 (time 1746075625.27 total 298.50 last 119.72) - open/close 41315 (time 1746075635.28 total 308.51 last 131.43) - open/close 42679 (time 1746075645.28 total 318.51 last 136.36) - open/close 44064 (time 1746075655.29 total 328.52 last 138.38) - open/close 45339 (time 1746075665.30 total 338.53 last 127.39) - open/close 46638 (time 1746075675.31 total 348.54 last 129.78) - open/close 47989 (time 1746075685.31 total 358.54 last 135.04) - open/close 49244 (time 1746075695.31 total 368.54 last 125.47) - open/close 50000 (time 1746075701.22 total 374.45 last 127.98) - open/close 51058 (time 1746075711.23 total 384.46 last 105.72) - open/close 52140 (time 1746075721.23 total 394.46 last 108.16) - open/close 53365 (time 1746075731.25 total 404.48 last 122.32) - open/close 54467 (time 1746075741.26 total 414.49 last 110.09) - open/close 55779 (time 1746075751.26 total 424.49 last 131.11) - open/close 57188 (time 1746075761.26 total 434.49 last 140.88) - open/close 58460 (time 1746075771.27 total 444.50 last 127.17) - open/close 59715 (time 1746075781.27 total 454.50 last 125.46) - open/close 60000 (time 1746075783.46 total 456.69 last 130.05) - open/close 61410 (time 1746075793.47 total 466.69 last 140.94) - open/close 62678 (time 1746075803.47 total 476.70 last 126.77) - open/close 64032 (time 1746075813.47 total 486.70 last 135.36) total: 64768 open/close in 491.59 seconds: 131.75 ops/second - /unlink 2437 (time 1746075829.84 total 10.01 last 243.45) - /unlink 4489 (time 1746075839.84 total 20.01 last 205.19) - /unlink 7220 (time 1746075849.84 total 30.01 last 273.05) - /unlink 9853 (time 1746075859.84 total 40.01 last 263.29) - /unlink 12443 (time 1746075869.85 total 50.01 last 258.95) - /unlink 15083 (time 1746075879.85 total 60.02 last 263.97) - /unlink 17788 (time 1746075889.85 total 70.02 last 270.35) - /unlink 20000 (time 1746075898.39 total 78.55 last 259.20) - /unlink 22470 (time 1746075908.39 total 88.56 last 246.96) - /unlink 24880 (time 1746075918.39 total 98.56 last 240.92) - /unlink 27482 (time 1746075928.39 total 108.56 last 260.18) - /unlink 30000 (time 1746075937.46 total 117.63 last 277.61) - /unlink 32986 (time 1746075947.46 total 127.63 last 298.59) - /unlink 35848 (time 1746075957.47 total 137.64 last 286.03) - /unlink 38634 (time 1746075967.47 total 147.64 last 278.46) - /unlink 40000 (time 1746075972.59 total 152.76 last 266.81) - /unlink 42694 (time 1746075982.60 total 162.77 last 269.32) - /unlink 45635 (time 1746075992.60 total 172.77 last 293.99) - /unlink 48177 (time 1746076002.60 total 182.77 last 254.14) - /unlink 50000 (time 1746076009.64 total 189.81 last 259.10) - /unlink 52431 (time 1746076019.64 total 199.81 last 243.03) - /unlink 54809 (time 1746076029.64 total 209.81 last 237.76) - /unlink 57278 (time 1746076039.64 total 219.81 last 246.88) - /unlink 59768 (time 1746076049.65 total 229.82 last 248.93) - /unlink 62360 (time 1746076059.65 total 239.82 last 259.14) total: 64768 /unlink in 249.48 seconds: 259.61 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg323-server unloading modules via unload_modules_local on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2