== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:18:00 (1746141480) Stopping clients: oleg217-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg217-client.virtnet /mnt/lustre2 (opts:-f) oleg217-server: oleg217-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions oleg217-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg217-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg217-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1551 (time 1746141532.72 total 10.00 last 155.05) - open/close 3333 (time 1746141542.72 total 20.01 last 178.12) - open/close 4661 (time 1746141552.73 total 30.01 last 132.78) - open/close 5998 (time 1746141562.73 total 40.02 last 133.61) - open/close 7218 (time 1746141572.74 total 50.02 last 121.92) - open/close 8434 (time 1746141582.74 total 60.02 last 121.57) - open/close 9646 (time 1746141592.74 total 70.03 last 121.18) - open/close 10000 (time 1746141595.48 total 72.77 last 129.21) - open/close 11253 (time 1746141605.48 total 82.77 last 125.29) - open/close 12734 (time 1746141615.49 total 92.77 last 148.02) - open/close 14282 (time 1746141625.50 total 102.78 last 154.69) - open/close 15607 (time 1746141635.50 total 112.78 last 132.45) - open/close 16833 (time 1746141645.50 total 122.78 last 122.59) - open/close 18079 (time 1746141655.50 total 132.79 last 124.55) - open/close 19214 (time 1746141665.51 total 142.79 last 113.43) - open/close 20000 (time 1746141671.83 total 149.12 last 124.30) - open/close 21255 (time 1746141681.84 total 159.12 last 125.45) - open/close 22530 (time 1746141691.84 total 169.12 last 127.46) - open/close 23788 (time 1746141701.85 total 179.13 last 125.75) - open/close 25067 (time 1746141711.85 total 189.13 last 127.83) - open/close 26119 (time 1746141721.86 total 199.14 last 105.09) - open/close 27425 (time 1746141731.87 total 209.15 last 130.50) - open/close 28420 (time 1746141741.87 total 219.15 last 99.48) - open/close 29559 (time 1746141751.87 total 229.15 last 113.90) - open/close 30000 (time 1746141756.07 total 233.35 last 105.08) - open/close 31180 (time 1746141766.07 total 243.36 last 117.93) - open/close 32199 (time 1746141776.07 total 253.36 last 101.90) - open/close 33262 (time 1746141786.08 total 263.36 last 106.28) - open/close 34183 (time 1746141796.08 total 273.36 last 92.05) - open/close 35148 (time 1746141806.08 total 283.37 last 96.48) - open/close 36153 (time 1746141816.09 total 293.37 last 100.43) - open/close 37110 (time 1746141826.09 total 303.38 last 95.67) - open/close 38042 (time 1746141836.10 total 313.38 last 93.15) - open/close 39057 (time 1746141846.11 total 323.39 last 101.41) - open/close 40000 (time 1746141856.13 total 333.41 last 94.08) - open/close 41001 (time 1746141866.14 total 343.42 last 100.03) - open/close 42000 (time 1746141876.14 total 353.42 last 99.87) - open/close 43125 (time 1746141886.15 total 363.43 last 112.36) - open/close 44244 (time 1746141896.16 total 373.44 last 111.87) - open/close 45370 (time 1746141906.16 total 383.44 last 112.59) - open/close 46762 (time 1746141916.16 total 393.44 last 139.19) - open/close 48021 (time 1746141926.16 total 403.44 last 125.86) - open/close 49147 (time 1746141936.17 total 413.45 last 112.54) - open/close 50000 (time 1746141943.85 total 421.13 last 111.05) - open/close 51036 (time 1746141953.85 total 431.14 last 103.53) - open/close 52049 (time 1746141963.86 total 441.14 last 101.23) - open/close 53201 (time 1746141973.86 total 451.15 last 115.17) - open/close 54216 (time 1746141983.86 total 461.15 last 101.49) - open/close 55380 (time 1746141993.87 total 471.15 last 116.34) - open/close 56518 (time 1746142003.87 total 481.16 last 113.74) - open/close 57597 (time 1746142013.88 total 491.16 last 107.86) - open/close 58653 (time 1746142023.89 total 501.17 last 105.50) - open/close 59861 (time 1746142033.89 total 511.17 last 120.79) - open/close 61163 (time 1746142043.89 total 521.18 last 130.13) - open/close 62250 (time 1746142053.90 total 531.18 last 108.63) - open/close 63581 (time 1746142063.90 total 541.18 last 133.10) total: 64768 open/close in 550.92 seconds: 117.56 ops/second - /unlink 2657 (time 1746142085.00 total 10.00 last 265.68) - /unlink 4844 (time 1746142095.00 total 20.00 last 218.68) - /unlink 6736 (time 1746142105.00 total 30.01 last 189.12) - /unlink 8536 (time 1746142115.01 total 40.01 last 179.86) - /unlink 10000 (time 1746142122.58 total 47.58 last 193.35) - /unlink 12511 (time 1746142132.59 total 57.59 last 250.95) - /unlink 15048 (time 1746142142.59 total 67.60 last 253.57) - /unlink 17403 (time 1746142152.60 total 77.60 last 235.43) - /unlink 19506 (time 1746142162.60 total 87.60 last 210.19) - /unlink 20000 (time 1746142165.06 total 90.06 last 201.26) - /unlink 22191 (time 1746142175.06 total 100.06 last 219.08) - /unlink 24089 (time 1746142185.06 total 110.06 last 189.78) - /unlink 25897 (time 1746142195.06 total 120.06 last 180.74) - /unlink 27969 (time 1746142205.07 total 130.07 last 207.05) - /unlink 30000 (time 1746142214.98 total 139.98 last 204.89) - /unlink 32021 (time 1746142224.99 total 149.99 last 202.02) - /unlink 34121 (time 1746142234.99 total 159.99 last 209.99) - /unlink 35949 (time 1746142244.99 total 169.99 last 182.71) - /unlink 37886 (time 1746142254.99 total 180.00 last 193.62) - /unlink 39813 (time 1746142265.00 total 190.00 last 192.69) - /unlink 41494 (time 1746142275.00 total 200.00 last 168.06) - /unlink 43103 (time 1746142285.00 total 210.00 last 160.88) - /unlink 44561 (time 1746142295.00 total 220.01 last 145.72) - /unlink 46056 (time 1746142305.01 total 230.02 last 149.36) - /unlink 47422 (time 1746142315.01 total 240.02 last 136.59) - /unlink 49232 (time 1746142325.02 total 250.03 last 180.82) - /unlink 50000 (time 1746142329.17 total 254.17 last 185.45) - /unlink 52050 (time 1746142339.17 total 264.17 last 204.94) - /unlink 54089 (time 1746142349.17 total 274.17 last 203.87) - /unlink 56004 (time 1746142359.17 total 284.17 last 191.48) - /unlink 58055 (time 1746142369.17 total 294.18 last 205.09) - /unlink 59929 (time 1746142379.18 total 304.18 last 187.33) - /unlink 61981 (time 1746142389.18 total 314.18 last 205.19) - /unlink 64195 (time 1746142399.18 total 324.18 last 221.40) total: 64768 /unlink in 327.21 seconds: 197.94 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1036 (time 1746142429.97 total 10.01 last 103.50) - open/close 2150 (time 1746142439.97 total 20.01 last 111.36) - open/close 3149 (time 1746142449.97 total 30.01 last 99.88) - open/close 4342 (time 1746142459.97 total 40.02 last 119.28) - open/close 5565 (time 1746142469.98 total 50.02 last 122.27) - open/close 6741 (time 1746142479.98 total 60.02 last 117.57) - open/close 8100 (time 1746142489.98 total 70.03 last 135.82) - open/close 9510 (time 1746142499.99 total 80.03 last 140.90) - open/close 10000 (time 1746142503.88 total 83.92 last 125.96) - open/close 11431 (time 1746142513.88 total 93.93 last 143.08) - open/close 12615 (time 1746142523.88 total 103.93 last 118.39) - open/close 13727 (time 1746142533.88 total 113.93 last 111.19) - open/close 14841 (time 1746142543.89 total 123.93 last 111.38) - open/close 15883 (time 1746142553.89 total 133.93 last 104.16) - open/close 17273 (time 1746142563.89 total 143.93 last 138.99) - open/close 18522 (time 1746142573.90 total 153.94 last 124.78) - open/close 19828 (time 1746142583.90 total 163.94 last 130.58) - open/close 21058 (time 1746142593.91 total 173.95 last 122.96) - open/close 22143 (time 1746142603.91 total 183.95 last 108.49) - open/close 23059 (time 1746142613.92 total 193.96 last 91.46) - open/close 23977 (time 1746142623.92 total 203.97 last 91.78) - open/close 24876 (time 1746142633.93 total 213.97 last 89.85) - open/close 26076 (time 1746142643.93 total 223.97 last 119.98) - open/close 27360 (time 1746142653.94 total 233.98 last 128.28) - open/close 28354 (time 1746142663.94 total 243.99 last 99.35) - open/close 29593 (time 1746142673.94 total 253.99 last 123.89) - open/close 30000 (time 1746142677.67 total 257.71 last 109.38) - open/close 31044 (time 1746142687.67 total 267.71 last 104.38) - open/close 32120 (time 1746142697.67 total 277.72 last 107.52) - open/close 33265 (time 1746142707.69 total 287.73 last 114.34) - open/close 34312 (time 1746142717.69 total 297.73 last 104.68) - open/close 35538 (time 1746142727.70 total 307.74 last 122.51) - open/close 36741 (time 1746142737.70 total 317.74 last 120.26) - open/close 37782 (time 1746142747.71 total 327.75 last 104.00) - open/close 38756 (time 1746142757.72 total 337.76 last 97.31) - open/close 39715 (time 1746142767.73 total 347.77 last 95.82) - open/close 40000 (time 1746142770.67 total 350.72 last 96.75) - open/close 41026 (time 1746142780.68 total 360.72 last 102.55) - open/close 42088 (time 1746142790.68 total 370.73 last 106.16) - open/close 43100 (time 1746142800.68 total 380.73 last 101.19) - open/close 44136 (time 1746142810.69 total 390.73 last 103.58) - open/close 45287 (time 1746142820.69 total 400.73 last 115.10) - open/close 46448 (time 1746142830.69 total 410.74 last 116.01) - open/close 47447 (time 1746142840.70 total 420.74 last 99.87) - open/close 48692 (time 1746142850.70 total 430.74 last 124.43) - open/close 49872 (time 1746142860.70 total 440.74 last 118.00) - open/close 51010 (time 1746142870.70 total 450.75 last 113.76) - open/close 52110 (time 1746142880.71 total 460.75 last 109.97) - open/close 53284 (time 1746142890.72 total 470.76 last 117.31) - open/close 54465 (time 1746142900.72 total 480.76 last 118.05) - open/close 55646 (time 1746142910.72 total 490.77 last 118.06) - open/close 56776 (time 1746142920.73 total 500.77 last 112.93) - open/close 57687 (time 1746142930.74 total 510.78 last 91.01) - open/close 58629 (time 1746142940.74 total 520.78 last 94.20) - open/close 59715 (time 1746142950.74 total 530.78 last 108.59) - open/close 60000 (time 1746142953.35 total 533.39 last 109.23) - open/close 61061 (time 1746142963.36 total 543.40 last 106.03) - open/close 62173 (time 1746142973.36 total 553.40 last 111.14) - open/close 63202 (time 1746142983.36 total 563.41 last 102.87) - open/close 64189 (time 1746142993.37 total 573.41 last 98.68) total: 64768 open/close in 579.97 seconds: 111.68 ops/second - /unlink 2258 (time 1746143011.72 total 10.00 last 225.75) - /unlink 4015 (time 1746143021.72 total 20.01 last 175.62) - /unlink 5611 (time 1746143031.73 total 30.01 last 159.51) - /unlink 7409 (time 1746143041.73 total 40.01 last 179.75) - /unlink 9118 (time 1746143051.73 total 50.02 last 170.88) - /unlink 10000 (time 1746143057.26 total 55.55 last 159.40) - /unlink 11605 (time 1746143067.26 total 65.55 last 160.48) - /unlink 13216 (time 1746143077.27 total 75.55 last 161.06) - /unlink 14603 (time 1746143087.27 total 85.55 last 138.69) - /unlink 16202 (time 1746143097.27 total 95.56 last 159.87) - /unlink 17855 (time 1746143107.28 total 105.56 last 165.18) - /unlink 19554 (time 1746143117.28 total 115.57 last 169.85) - /unlink 20000 (time 1746143120.05 total 118.34 last 160.78) - /unlink 21605 (time 1746143130.06 total 128.35 last 160.39) - /unlink 23263 (time 1746143140.06 total 138.35 last 165.75) - /unlink 24821 (time 1746143150.07 total 148.36 last 155.64) - /unlink 26363 (time 1746143160.08 total 158.37 last 154.08) - /unlink 27840 (time 1746143170.09 total 168.37 last 147.62) - /unlink 29321 (time 1746143180.13 total 178.42 last 147.44) - /unlink 30000 (time 1746143184.23 total 182.52 last 165.55) - /unlink 31875 (time 1746143194.24 total 192.52 last 187.40) - /unlink 33435 (time 1746143204.24 total 202.53 last 155.96) - /unlink 35077 (time 1746143214.24 total 212.53 last 164.16) - /unlink 36772 (time 1746143224.25 total 222.53 last 169.47) - /unlink 38621 (time 1746143234.25 total 232.53 last 184.84) - /unlink 40000 (time 1746143242.25 total 240.53 last 172.43) - /unlink 41563 (time 1746143252.25 total 250.54 last 156.23) - /unlink 43327 (time 1746143262.25 total 260.54 last 176.39) - /unlink 45067 (time 1746143272.26 total 270.55 last 173.86) - /unlink 46693 (time 1746143282.27 total 280.55 last 162.50) - /unlink 48422 (time 1746143292.27 total 290.56 last 172.83) - /unlink 50000 (time 1746143301.04 total 299.32 last 180.01) - /unlink 51496 (time 1746143311.04 total 309.33 last 149.54) - /unlink 53260 (time 1746143321.04 total 319.33 last 176.31) - /unlink 54815 (time 1746143331.05 total 329.34 last 155.41) - /unlink 56461 (time 1746143341.06 total 339.34 last 164.50) - /unlink 57998 (time 1746143351.06 total 349.35 last 153.63) - /unlink 59681 (time 1746143361.07 total 359.35 last 168.19) - /unlink 61406 (time 1746143371.09 total 369.37 last 172.14) - /unlink 63116 (time 1746143381.09 total 379.38 last 170.92) total: 64768 /unlink in 388.55 seconds: 166.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 903 (time 1746143421.54 total 10.01 last 90.24) - open/close 1920 (time 1746143431.56 total 20.02 last 101.55) - open/close 2994 (time 1746143441.56 total 30.03 last 107.36) - open/close 4131 (time 1746143451.57 total 40.03 last 113.68) - open/close 5204 (time 1746143461.57 total 50.04 last 107.22) - open/close 6295 (time 1746143471.58 total 60.04 last 109.04) - open/close 7440 (time 1746143481.58 total 70.05 last 114.45) - open/close 8515 (time 1746143491.58 total 80.05 last 107.50) - open/close 9598 (time 1746143501.59 total 90.05 last 108.26) - open/close 10000 (time 1746143505.39 total 93.85 last 105.78) - open/close 11060 (time 1746143515.39 total 103.85 last 105.96) - open/close 12140 (time 1746143525.39 total 113.86 last 107.96) - open/close 13291 (time 1746143535.39 total 123.86 last 115.09) - open/close 14393 (time 1746143545.40 total 133.86 last 110.16) - open/close 15406 (time 1746143555.41 total 143.87 last 101.22) - open/close 16328 (time 1746143565.41 total 153.87 last 92.20) - open/close 17312 (time 1746143575.41 total 163.87 last 98.34) - open/close 18383 (time 1746143585.42 total 173.88 last 107.05) - open/close 19407 (time 1746143595.42 total 183.88 last 102.39) - open/close 20000 (time 1746143601.32 total 189.78 last 100.47) - open/close 21079 (time 1746143611.33 total 199.79 last 107.83) - open/close 22141 (time 1746143621.33 total 209.80 last 106.13) - open/close 23126 (time 1746143631.33 total 219.80 last 98.50) - open/close 24248 (time 1746143641.33 total 229.80 last 112.19) - open/close 25290 (time 1746143651.34 total 239.80 last 104.14) - open/close 26338 (time 1746143661.35 total 249.82 last 104.65) - open/close 27373 (time 1746143671.36 total 259.82 last 103.43) - open/close 28396 (time 1746143681.36 total 269.82 last 102.28) - open/close 29356 (time 1746143691.37 total 279.83 last 95.94) - open/close 30000 (time 1746143698.10 total 286.56 last 95.64) - open/close 31086 (time 1746143708.11 total 296.57 last 108.56) - open/close 32240 (time 1746143718.11 total 306.57 last 115.40) - open/close 33285 (time 1746143728.12 total 316.58 last 104.39) - open/close 34318 (time 1746143738.12 total 326.58 last 103.29) - open/close 35403 (time 1746143748.12 total 336.58 last 108.50) - open/close 36495 (time 1746143758.13 total 346.59 last 109.09) - open/close 37716 (time 1746143768.13 total 356.59 last 122.07) - open/close 38949 (time 1746143778.14 total 366.60 last 123.16) - open/close 40000 (time 1746143787.05 total 375.52 last 117.94) - open/close 41181 (time 1746143797.05 total 385.52 last 118.09) - open/close 42492 (time 1746143807.06 total 395.52 last 131.06) - open/close 43825 (time 1746143817.06 total 405.53 last 133.21) - open/close 45165 (time 1746143827.07 total 415.53 last 133.96) - open/close 46323 (time 1746143837.07 total 425.53 last 115.76) - open/close 47463 (time 1746143847.07 total 435.54 last 113.96) - open/close 48582 (time 1746143857.08 total 445.54 last 111.89) - open/close 49734 (time 1746143867.09 total 455.55 last 115.03) - open/close 50000 (time 1746143869.67 total 458.13 last 103.09) - open/close 51257 (time 1746143879.68 total 468.14 last 125.63) - open/close 52652 (time 1746143889.68 total 478.14 last 139.49) - open/close 53958 (time 1746143899.68 total 488.14 last 130.59) - open/close 55375 (time 1746143909.68 total 498.15 last 141.62) - open/close 56538 (time 1746143919.68 total 508.15 last 116.30) - open/close 57646 (time 1746143929.69 total 518.15 last 110.75) - open/close 58805 (time 1746143939.69 total 528.16 last 115.83) - open/close 59980 (time 1746143949.70 total 538.16 last 117.47) - open/close 61090 (time 1746143959.70 total 548.16 last 110.98) - open/close 62391 (time 1746143969.70 total 558.16 last 130.06) - open/close 63481 (time 1746143979.70 total 568.17 last 108.97) total: 64768 open/close in 577.90 seconds: 112.07 ops/second - /unlink 2202 (time 1746144001.16 total 10.01 last 220.01) - /unlink 4368 (time 1746144011.18 total 20.02 last 216.26) - /unlink 6155 (time 1746144021.18 total 30.03 last 178.61) - /unlink 8008 (time 1746144031.19 total 40.04 last 185.16) - /unlink 9803 (time 1746144041.19 total 50.04 last 179.38) - /unlink 11647 (time 1746144051.20 total 60.05 last 184.36) - /unlink 13325 (time 1746144061.20 total 70.05 last 167.76) - /unlink 14821 (time 1746144071.21 total 80.06 last 149.49) - /unlink 16482 (time 1746144081.21 total 90.06 last 166.03) - /unlink 18237 (time 1746144091.21 total 100.06 last 175.49) - /unlink 19918 (time 1746144101.23 total 110.08 last 167.79) - /unlink 21823 (time 1746144111.26 total 120.11 last 189.86) - /unlink 23821 (time 1746144121.27 total 130.12 last 199.74) - /unlink 25639 (time 1746144131.27 total 140.12 last 181.70) - /unlink 27529 (time 1746144141.28 total 150.13 last 188.82) - /unlink 29599 (time 1746144151.28 total 160.13 last 206.96) - /unlink 31698 (time 1746144161.29 total 170.14 last 209.82) - /unlink 33548 (time 1746144171.29 total 180.14 last 184.94) - /unlink 35480 (time 1746144181.29 total 190.14 last 193.16) - /unlink 37458 (time 1746144191.29 total 200.14 last 197.78) - /unlink 39589 (time 1746144201.30 total 210.14 last 213.05) - /unlink 41465 (time 1746144211.30 total 220.15 last 187.58) - /unlink 43367 (time 1746144221.30 total 230.15 last 190.19) - /unlink 45236 (time 1746144231.30 total 240.15 last 186.89) - /unlink 47282 (time 1746144241.30 total 250.15 last 204.58) - /unlink 49654 (time 1746144251.30 total 260.15 last 237.19) - /unlink 51682 (time 1746144261.30 total 270.15 last 202.76) - /unlink 53620 (time 1746144271.30 total 280.15 last 193.73) - /unlink 55762 (time 1746144281.31 total 290.16 last 214.16) - /unlink 57861 (time 1746144291.31 total 300.16 last 209.81) - /unlink 59725 (time 1746144301.31 total 310.16 last 186.34) - /unlink 61490 (time 1746144311.32 total 320.17 last 176.45) - /unlink 63320 (time 1746144321.32 total 330.17 last 182.95) total: 64768 /unlink in 336.06 seconds: 192.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1395 (time 1746144357.78 total 10.01 last 139.41) - open/close 2762 (time 1746144367.78 total 20.01 last 136.70) - open/close 3989 (time 1746144377.79 total 30.01 last 122.63) - open/close 5158 (time 1746144387.80 total 40.02 last 116.83) - open/close 6370 (time 1746144397.80 total 50.02 last 121.17) - open/close 7730 (time 1746144407.80 total 60.03 last 135.92) - open/close 9073 (time 1746144417.81 total 70.04 last 134.17) - open/close 10000 (time 1746144424.94 total 77.16 last 130.15) - open/close 11251 (time 1746144434.94 total 87.16 last 125.05) - open/close 12368 (time 1746144444.94 total 97.17 last 111.66) - open/close 13593 (time 1746144454.95 total 107.17 last 122.43) - open/close 14966 (time 1746144464.95 total 117.18 last 137.26) - open/close 16511 (time 1746144474.95 total 127.18 last 154.50) - open/close 18243 (time 1746144484.96 total 137.18 last 173.16) - open/close 20000 (time 1746144492.74 total 144.97 last 225.59) - open/close 22254 (time 1746144502.75 total 154.97 last 225.27) - open/close 24156 (time 1746144512.75 total 164.97 last 190.16) - open/close 25777 (time 1746144522.76 total 174.98 last 162.00) - open/close 27280 (time 1746144532.76 total 184.98 last 150.29) - open/close 28997 (time 1746144542.77 total 194.99 last 171.57) - open/close 30000 (time 1746144549.06 total 201.29 last 159.30) - open/close 31501 (time 1746144559.06 total 211.29 last 150.09) - open/close 33178 (time 1746144569.07 total 221.29 last 167.61) - open/close 34607 (time 1746144579.07 total 231.29 last 142.87) - open/close 36059 (time 1746144589.08 total 241.30 last 145.14) - open/close 37448 (time 1746144599.08 total 251.30 last 138.87) - open/close 38997 (time 1746144609.08 total 261.30 last 154.88) - open/close 40000 (time 1746144616.22 total 268.44 last 140.42) - open/close 41633 (time 1746144626.22 total 278.45 last 163.27) - open/close 43210 (time 1746144636.23 total 288.45 last 157.64) - open/close 44877 (time 1746144646.23 total 298.46 last 166.59) - open/close 46956 (time 1746144656.24 total 308.46 last 207.79) - open/close 48333 (time 1746144666.24 total 318.46 last 137.66) - open/close 49688 (time 1746144676.25 total 328.47 last 135.38) - open/close 50000 (time 1746144678.44 total 330.66 last 142.36) - open/close 51452 (time 1746144688.45 total 340.67 last 145.14) - open/close 52950 (time 1746144698.46 total 350.68 last 149.67) - open/close 54304 (time 1746144708.46 total 360.68 last 135.40) - open/close 55675 (time 1746144718.46 total 370.68 last 137.02) - open/close 57196 (time 1746144728.47 total 380.69 last 151.98) - open/close 58818 (time 1746144738.47 total 390.70 last 162.15) - open/close 60000 (time 1746144745.25 total 397.47 last 174.53) - open/close 61506 (time 1746144755.25 total 407.47 last 150.54) - open/close 62768 (time 1746144765.25 total 417.48 last 126.14) - open/close 64279 (time 1746144775.26 total 427.48 last 150.98) total: 64768 open/close in 430.83 seconds: 150.33 ops/second - /unlink 2661 (time 1746144790.12 total 10.00 last 266.00) - /unlink 5247 (time 1746144800.12 total 20.01 last 258.55) - /unlink 7397 (time 1746144810.12 total 30.01 last 214.92) - /unlink 9483 (time 1746144820.13 total 40.01 last 208.49) - /unlink 10000 (time 1746144822.33 total 42.22 last 234.58) - /unlink 12142 (time 1746144832.33 total 52.22 last 214.13) - /unlink 14725 (time 1746144842.34 total 62.22 last 258.29) - /unlink 17173 (time 1746144852.34 total 72.22 last 244.75) - /unlink 19338 (time 1746144862.34 total 82.23 last 216.38) - /unlink 20000 (time 1746144866.36 total 86.25 last 164.78) - /unlink 21946 (time 1746144876.36 total 96.25 last 194.56) - /unlink 24118 (time 1746144886.36 total 106.25 last 217.17) - /unlink 26114 (time 1746144896.37 total 116.25 last 199.55) - /unlink 28539 (time 1746144906.37 total 126.25 last 242.45) - /unlink 30000 (time 1746144912.38 total 132.27 last 243.03) - /unlink 32047 (time 1746144922.38 total 142.27 last 204.70) - /unlink 34142 (time 1746144932.38 total 152.27 last 209.47) - /unlink 36340 (time 1746144942.38 total 162.27 last 219.78) - /unlink 38716 (time 1746144952.38 total 172.27 last 237.60) - /unlink 40000 (time 1746144957.56 total 177.45 last 247.97) - /unlink 42228 (time 1746144967.56 total 187.45 last 222.79) - /unlink 44372 (time 1746144977.56 total 197.45 last 214.35) - /unlink 46560 (time 1746144987.56 total 207.45 last 218.75) - /unlink 48777 (time 1746144997.57 total 217.45 last 221.67) - /unlink 50000 (time 1746145003.74 total 223.62 last 198.19) - /unlink 52374 (time 1746145013.74 total 233.63 last 237.31) - /unlink 54659 (time 1746145023.74 total 243.63 last 228.43) - /unlink 56866 (time 1746145033.75 total 253.64 last 220.54) - /unlink 58969 (time 1746145043.75 total 263.64 last 210.28) - /unlink 60000 (time 1746145048.45 total 268.34 last 219.34) - /unlink 62155 (time 1746145058.46 total 278.34 last 215.43) - /unlink 64498 (time 1746145068.46 total 288.35 last 234.21) total: 64768 /unlink in 289.57 seconds: 223.67 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1572 (time 1746145098.79 total 10.01 last 157.11) - open/close 3069 (time 1746145108.79 total 20.01 last 149.68) - open/close 4434 (time 1746145118.79 total 30.01 last 136.40) - open/close 5955 (time 1746145128.80 total 40.02 last 152.09) - open/close 7535 (time 1746145138.80 total 50.02 last 157.95) - open/close 9036 (time 1746145148.80 total 60.02 last 150.07) - open/close 10000 (time 1746145154.65 total 65.87 last 164.84) - open/close 11491 (time 1746145164.66 total 75.88 last 148.86) - open/close 12856 (time 1746145174.67 total 85.89 last 136.44) - open/close 14399 (time 1746145184.67 total 95.89 last 154.30) - open/close 16289 (time 1746145194.67 total 105.89 last 188.98) - open/close 18175 (time 1746145204.67 total 115.89 last 188.60) - open/close 19859 (time 1746145214.67 total 125.89 last 168.37) - open/close 21235 (time 1746145224.68 total 135.90 last 137.55) - open/close 22813 (time 1746145234.68 total 145.90 last 157.67) - open/close 24628 (time 1746145244.69 total 155.91 last 181.44) - open/close 26047 (time 1746145254.69 total 165.91 last 141.85) - open/close 27552 (time 1746145264.69 total 175.91 last 150.50) - open/close 29161 (time 1746145274.70 total 185.92 last 160.79) - open/close 30000 (time 1746145280.44 total 191.66 last 146.15) - open/close 31381 (time 1746145290.44 total 201.66 last 138.06) - open/close 32683 (time 1746145300.45 total 211.67 last 130.14) - open/close 34148 (time 1746145310.45 total 221.67 last 146.43) - open/close 35712 (time 1746145320.45 total 231.67 last 156.36) - open/close 37380 (time 1746145330.46 total 241.68 last 166.76) - open/close 39156 (time 1746145340.46 total 251.68 last 177.55) - open/close 40000 (time 1746145345.96 total 257.18 last 153.31) - open/close 41549 (time 1746145355.97 total 267.19 last 154.87) - open/close 43078 (time 1746145365.97 total 277.19 last 152.80) - open/close 44699 (time 1746145375.98 total 287.20 last 162.01) - open/close 46359 (time 1746145385.98 total 297.20 last 165.99) - open/close 47881 (time 1746145395.98 total 307.20 last 152.19) - open/close 49671 (time 1746145405.99 total 317.20 last 178.90) - open/close 50000 (time 1746145408.09 total 319.31 last 156.21) - open/close 51396 (time 1746145418.09 total 329.31 last 139.56) - open/close 52873 (time 1746145428.10 total 339.32 last 147.68) - open/close 54330 (time 1746145438.10 total 349.32 last 145.66) - open/close 56172 (time 1746145448.10 total 359.32 last 184.19) - open/close 57770 (time 1746145458.10 total 369.32 last 159.75) - open/close 59224 (time 1746145468.11 total 379.33 last 145.34) - open/close 60000 (time 1746145473.40 total 384.62 last 146.49) - open/close 61562 (time 1746145483.41 total 394.62 last 156.17) - open/close 63062 (time 1746145493.42 total 404.64 last 149.84) - open/close 64528 (time 1746145503.42 total 414.64 last 146.47) total: 64768 open/close in 416.35 seconds: 155.56 ops/second - /unlink 2101 (time 1746145516.75 total 10.00 last 210.08) - /unlink 4315 (time 1746145526.75 total 20.00 last 221.37) - /unlink 6319 (time 1746145536.75 total 30.00 last 200.38) - /unlink 8073 (time 1746145546.75 total 40.01 last 175.35) - /unlink 9935 (time 1746145556.75 total 50.01 last 186.17) - /unlink 12374 (time 1746145566.76 total 60.01 last 243.86) - /unlink 14422 (time 1746145576.76 total 70.01 last 204.78) - /unlink 16924 (time 1746145586.76 total 80.01 last 250.17) - /unlink 19318 (time 1746145596.76 total 90.01 last 239.33) - /unlink 20000 (time 1746145601.01 total 94.26 last 160.63) - /unlink 22016 (time 1746145611.01 total 104.26 last 201.53) - /unlink 24031 (time 1746145621.01 total 114.26 last 201.48) - /unlink 26241 (time 1746145631.01 total 124.26 last 220.98) - /unlink 28556 (time 1746145641.01 total 134.27 last 231.44) - /unlink 30000 (time 1746145647.63 total 140.88 last 218.39) - /unlink 32330 (time 1746145657.63 total 150.88 last 232.97) - /unlink 34850 (time 1746145667.63 total 160.88 last 251.91) - /unlink 37078 (time 1746145677.64 total 170.89 last 222.70) - /unlink 39129 (time 1746145687.64 total 180.89 last 205.05) - /unlink 40000 (time 1746145690.84 total 184.09 last 272.38) - /unlink 42166 (time 1746145700.84 total 194.09 last 216.55) - /unlink 44602 (time 1746145710.84 total 204.09 last 243.51) - /unlink 47158 (time 1746145720.84 total 214.10 last 255.56) - /unlink 49587 (time 1746145730.84 total 224.10 last 242.89) - /unlink 51978 (time 1746145740.84 total 234.10 last 239.09) - /unlink 54395 (time 1746145750.84 total 244.10 last 241.69) - /unlink 56629 (time 1746145760.84 total 254.10 last 223.39) - /unlink 58686 (time 1746145770.85 total 264.10 last 205.64) - /unlink 60000 (time 1746145777.67 total 270.92 last 192.65) - /unlink 62340 (time 1746145787.67 total 280.92 last 233.92) - /unlink 64678 (time 1746145797.67 total 290.93 last 233.75) total: 64768 /unlink in 291.39 seconds: 222.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server unloading modules via unload_modules_local on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2