== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:57:47 (1769079467) Stopping clients: oleg140-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg140-client.virtnet /mnt/lustre2 (opts:-f) oleg140-server: oleg140-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg140-server: detected 4 online CPUs by sysfs oleg140-server: MODOPTS_LIBCFS= oleg140-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg140-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg140-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 840 (time 1769079598.21 total 10.00 last 83.99) - open/close 1597 (time 1769079608.21 total 20.01 last 75.65) - open/close 2400 (time 1769079618.23 total 30.03 last 80.14) - open/close 3231 (time 1769079628.23 total 40.03 last 83.10) - open/close 3961 (time 1769079638.25 total 50.05 last 72.86) - open/close 4691 (time 1769079648.27 total 60.06 last 72.91) - open/close 5456 (time 1769079658.28 total 70.07 last 76.42) - open/close 6241 (time 1769079668.29 total 80.08 last 78.39) - open/close 7048 (time 1769079678.30 total 90.09 last 80.65) - open/close 7848 (time 1769079688.30 total 100.09 last 79.97) - open/close 8631 (time 1769079698.30 total 110.09 last 78.29) - open/close 9421 (time 1769079708.30 total 120.09 last 78.99) - open/close 10000 (time 1769079715.61 total 127.40 last 79.27) - open/close 10863 (time 1769079725.62 total 137.41 last 86.21) - open/close 11624 (time 1769079735.63 total 147.42 last 76.01) - open/close 12353 (time 1769079745.63 total 157.42 last 72.90) - open/close 13187 (time 1769079755.64 total 167.43 last 83.28) - open/close 14040 (time 1769079765.64 total 177.44 last 85.29) - open/close 14842 (time 1769079775.65 total 187.45 last 80.13) - open/close 15633 (time 1769079785.68 total 197.47 last 78.92) - open/close 16406 (time 1769079795.68 total 207.47 last 77.27) - open/close 17249 (time 1769079805.68 total 217.48 last 84.27) - open/close 18131 (time 1769079815.68 total 227.48 last 88.19) - open/close 19027 (time 1769079825.68 total 237.48 last 89.60) - open/close 19874 (time 1769079835.69 total 247.49 last 84.63) - open/close 20660 (time 1769079845.72 total 257.51 last 78.37) - open/close 21474 (time 1769079855.73 total 267.52 last 81.37) - open/close 22275 (time 1769079865.73 total 277.52 last 80.08) - open/close 23101 (time 1769079875.73 total 287.53 last 82.55) - open/close 23894 (time 1769079885.74 total 297.53 last 79.27) - open/close 24693 (time 1769079895.74 total 307.54 last 79.87) - open/close 25580 (time 1769079905.75 total 317.54 last 88.67) - open/close 26509 (time 1769079915.76 total 327.55 last 92.77) - open/close 27268 (time 1769079925.76 total 337.56 last 75.87) - open/close 28161 (time 1769079935.77 total 347.56 last 89.28) - open/close 29036 (time 1769079945.77 total 357.56 last 87.48) - open/close 29831 (time 1769079955.79 total 367.58 last 79.34) - open/close 30000 (time 1769079958.04 total 369.83 last 75.23) - open/close 30868 (time 1769079968.04 total 379.83 last 86.79) - open/close 31731 (time 1769079978.07 total 389.86 last 85.99) - open/close 32456 (time 1769079988.08 total 399.87 last 72.43) - open/close 33222 (time 1769079998.09 total 409.88 last 76.53) - open/close 34076 (time 1769080008.10 total 419.89 last 85.35) - open/close 34804 (time 1769080018.10 total 429.89 last 72.76) - open/close 35616 (time 1769080028.11 total 439.90 last 81.17) - open/close 36511 (time 1769080038.11 total 449.90 last 89.48) - open/close 37397 (time 1769080048.11 total 459.90 last 88.58) - open/close 38251 (time 1769080058.11 total 469.91 last 85.37) - open/close 39092 (time 1769080068.12 total 479.91 last 84.03) - open/close 39897 (time 1769080078.12 total 489.91 last 80.49) - open/close 40708 (time 1769080088.12 total 499.92 last 81.10) - open/close 41616 (time 1769080098.13 total 509.92 last 90.75) - open/close 42513 (time 1769080108.14 total 519.93 last 89.60) - open/close 43334 (time 1769080118.15 total 529.94 last 82.02) - open/close 44209 (time 1769080128.15 total 539.94 last 87.48) - open/close 45089 (time 1769080138.16 total 549.95 last 87.97) - open/close 45823 (time 1769080148.16 total 559.95 last 73.37) - open/close 46705 (time 1769080158.17 total 569.96 last 88.13) - open/close 47555 (time 1769080168.18 total 579.97 last 84.93) - open/close 48401 (time 1769080178.19 total 589.98 last 84.50) - open/close 49283 (time 1769080188.19 total 599.98 last 88.17) - open/close 50000 (time 1769080196.15 total 607.94 last 90.09) - open/close 50827 (time 1769080206.15 total 617.95 last 82.67) - open/close 51687 (time 1769080216.16 total 627.95 last 85.97) - open/close 52520 (time 1769080226.16 total 637.95 last 83.28) - open/close 53369 (time 1769080236.17 total 647.96 last 84.84) - open/close 54176 (time 1769080246.18 total 657.97 last 80.62) - open/close 55017 (time 1769080256.18 total 667.97 last 84.09) - open/close 55784 (time 1769080266.19 total 677.98 last 76.59) - open/close 56687 (time 1769080276.21 total 688.00 last 90.16) - open/close 57490 (time 1769080286.21 total 698.00 last 80.25) - open/close 58261 (time 1769080296.22 total 708.01 last 77.05) - open/close 59066 (time 1769080306.22 total 718.01 last 80.49) - open/close 59915 (time 1769080316.23 total 728.02 last 84.84) - open/close 60667 (time 1769080326.23 total 738.02 last 75.19) - open/close 61501 (time 1769080336.23 total 748.02 last 83.38) - open/close 62384 (time 1769080346.23 total 758.02 last 88.29) - open/close 63223 (time 1769080356.24 total 768.03 last 83.83) - open/close 64051 (time 1769080366.24 total 778.04 last 82.78) total: 64768 open/close in 785.84 seconds: 82.42 ops/second - /unlink 737 (time 1769080387.73 total 10.00 last 73.68) - /unlink 1548 (time 1769080397.73 total 20.00 last 81.08) - /unlink 2382 (time 1769080407.74 total 30.01 last 83.32) - /unlink 3129 (time 1769080417.75 total 40.02 last 74.65) - /unlink 3867 (time 1769080427.76 total 50.03 last 73.75) - /unlink 4711 (time 1769080437.77 total 60.04 last 84.33) - /unlink 5700 (time 1769080447.77 total 70.04 last 98.85) - /unlink 6610 (time 1769080457.77 total 80.04 last 90.97) - /unlink 7573 (time 1769080467.78 total 90.05 last 96.26) - /unlink 8439 (time 1769080477.78 total 100.05 last 86.56) - /unlink 9298 (time 1769080487.80 total 110.07 last 85.78) - /unlink 10000 (time 1769080495.56 total 117.83 last 90.45) - /unlink 11015 (time 1769080505.56 total 127.83 last 101.48) - /unlink 11897 (time 1769080515.57 total 137.84 last 88.10) - /unlink 12827 (time 1769080525.57 total 147.84 last 92.98) - /unlink 13854 (time 1769080535.58 total 157.85 last 102.60) - /unlink 14883 (time 1769080545.59 total 167.86 last 102.82) - /unlink 15818 (time 1769080555.60 total 177.87 last 93.38) - /unlink 16841 (time 1769080565.61 total 187.88 last 102.23) - /unlink 17909 (time 1769080575.61 total 197.88 last 106.76) - /unlink 18850 (time 1769080585.62 total 207.89 last 94.06) - /unlink 19877 (time 1769080595.62 total 217.89 last 102.69) - /unlink 20929 (time 1769080605.63 total 227.90 last 105.13) - /unlink 21931 (time 1769080615.63 total 237.90 last 100.18) - /unlink 22963 (time 1769080625.63 total 247.90 last 103.18) - /unlink 24053 (time 1769080635.63 total 257.91 last 108.93) - /unlink 25031 (time 1769080645.64 total 267.91 last 97.72) - /unlink 26137 (time 1769080655.64 total 277.91 last 110.59) - /unlink 27168 (time 1769080665.65 total 287.92 last 103.05) - /unlink 28277 (time 1769080675.65 total 297.92 last 110.84) - /unlink 29383 (time 1769080685.66 total 307.93 last 110.56) - /unlink 30000 (time 1769080691.57 total 313.84 last 104.31) - /unlink 30994 (time 1769080701.58 total 323.85 last 99.37) - /unlink 32030 (time 1769080711.58 total 333.85 last 103.55) - /unlink 33120 (time 1769080721.58 total 343.85 last 108.98) - /unlink 34065 (time 1769080731.59 total 353.86 last 94.40) - /unlink 35142 (time 1769080741.60 total 363.87 last 107.65) - /unlink 36261 (time 1769080751.60 total 373.87 last 111.90) - /unlink 37280 (time 1769080761.61 total 383.88 last 101.77) - /unlink 38335 (time 1769080771.62 total 393.89 last 105.37) - /unlink 39460 (time 1769080781.63 total 403.90 last 112.43) - /unlink 40000 (time 1769080786.50 total 408.77 last 110.89) - /unlink 41276 (time 1769080796.50 total 418.77 last 127.56) - /unlink 42450 (time 1769080806.51 total 428.78 last 117.30) - /unlink 43606 (time 1769080816.51 total 438.79 last 115.55) - /unlink 44735 (time 1769080826.52 total 448.79 last 112.87) - /unlink 46091 (time 1769080836.53 total 458.80 last 135.45) - /unlink 47355 (time 1769080846.53 total 468.80 last 126.39) - /unlink 48393 (time 1769080856.53 total 478.80 last 103.79) - /unlink 49468 (time 1769080866.54 total 488.81 last 107.43) - /unlink 50000 (time 1769080871.72 total 493.99 last 102.65) - /unlink 51099 (time 1769080881.73 total 504.00 last 109.82) - /unlink 52127 (time 1769080891.73 total 514.01 last 102.72) - /unlink 53268 (time 1769080901.74 total 524.01 last 114.08) - /unlink 54317 (time 1769080911.74 total 534.01 last 104.89) - /unlink 55388 (time 1769080921.74 total 544.01 last 107.09) - /unlink 56410 (time 1769080931.74 total 554.01 last 102.14) - /unlink 57465 (time 1769080941.75 total 564.02 last 105.47) - /unlink 58501 (time 1769080951.75 total 574.02 last 103.59) - /unlink 59661 (time 1769080961.75 total 584.03 last 115.92) - /unlink 60000 (time 1769080964.79 total 587.07 last 111.52) - /unlink 61084 (time 1769080974.80 total 597.07 last 108.37) - /unlink 62191 (time 1769080984.81 total 607.08 last 110.61) - /unlink 63565 (time 1769080994.82 total 617.09 last 137.25) total: 64768 /unlink in 626.84 seconds: 103.32 ops/second Waiting for MDT destroys to complete oleg140-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 - open/close 1177 (time 1769081025.24 total 10.00 last 117.66) - open/close 2219 (time 1769081035.24 total 20.00 last 104.20) - open/close 3235 (time 1769081045.25 total 30.00 last 101.59) - open/close 4165 (time 1769081055.26 total 40.02 last 92.90) - open/close 4654 (time 1769081065.26 total 50.02 last 48.89) - open/close 5197 (time 1769081075.27 total 60.03 last 54.24) - open/close 5548 (time 1769081085.28 total 70.04 last 35.07) - open/close 5927 (time 1769081095.34 total 80.10 last 37.67) - open/close 6448 (time 1769081105.34 total 90.10 last 52.09) - open/close 7106 (time 1769081115.34 total 100.10 last 65.79) - open/close 7876 (time 1769081125.37 total 110.13 last 76.80) - open/close 8664 (time 1769081135.38 total 120.14 last 78.70) - open/close 9448 (time 1769081145.41 total 130.17 last 78.16) - open/close 10000 (time 1769081151.73 total 136.49 last 87.41) - open/close 10803 (time 1769081161.75 total 146.51 last 80.14) - open/close 11584 (time 1769081171.77 total 156.53 last 77.93) - open/close 12431 (time 1769081181.77 total 166.53 last 84.70) - open/close 13282 (time 1769081191.78 total 176.54 last 85.01) - open/close 14041 (time 1769081201.79 total 186.55 last 75.81) - open/close 14829 (time 1769081211.80 total 196.56 last 78.72) - open/close 15642 (time 1769081221.81 total 206.57 last 81.24) - open/close 16459 (time 1769081231.81 total 216.57 last 81.68) - open/close 17253 (time 1769081241.82 total 226.58 last 79.37) - open/close 18170 (time 1769081251.82 total 236.58 last 91.66) - open/close 18960 (time 1769081261.82 total 246.58 last 78.98) - open/close 19796 (time 1769081271.82 total 256.58 last 83.59) - open/close 20000 (time 1769081274.09 total 258.85 last 89.99) - open/close 20886 (time 1769081284.10 total 268.86 last 88.53) - open/close 21723 (time 1769081294.11 total 278.86 last 83.65) - open/close 22495 (time 1769081304.11 total 288.87 last 77.20) - open/close 23367 (time 1769081314.11 total 298.87 last 87.17) - open/close 24063 (time 1769081324.12 total 308.88 last 69.55) - open/close 24793 (time 1769081334.13 total 318.89 last 72.91) - open/close 25610 (time 1769081344.13 total 328.89 last 81.65) - open/close 26453 (time 1769081354.14 total 338.90 last 84.23) - open/close 27243 (time 1769081364.16 total 348.92 last 78.87) - open/close 27929 (time 1769081374.17 total 358.93 last 68.54) - open/close 28663 (time 1769081384.18 total 368.94 last 73.32) - open/close 29372 (time 1769081394.18 total 378.94 last 70.88) - open/close 30000 (time 1769081402.09 total 386.85 last 79.38) - open/close 30894 (time 1769081412.10 total 396.86 last 89.35) - open/close 31727 (time 1769081422.11 total 406.87 last 83.23) - open/close 32545 (time 1769081432.11 total 416.87 last 81.78) - open/close 33451 (time 1769081442.13 total 426.89 last 90.47) - open/close 34251 (time 1769081452.14 total 436.90 last 79.91) - open/close 35055 (time 1769081462.14 total 446.90 last 80.37) - open/close 35870 (time 1769081472.15 total 456.91 last 81.43) - open/close 36699 (time 1769081482.15 total 466.91 last 82.90) - open/close 37541 (time 1769081492.17 total 476.92 last 84.07) - open/close 38348 (time 1769081502.17 total 486.93 last 80.69) - open/close 39165 (time 1769081512.18 total 496.94 last 81.59) - open/close 40000 (time 1769081521.64 total 506.40 last 88.29) - open/close 40877 (time 1769081531.64 total 516.40 last 87.69) - open/close 41749 (time 1769081541.64 total 526.40 last 87.20) - open/close 42576 (time 1769081551.65 total 536.41 last 82.64) - open/close 43356 (time 1769081561.66 total 546.42 last 77.86) - open/close 44237 (time 1769081571.69 total 556.45 last 87.89) - open/close 45064 (time 1769081581.70 total 566.46 last 82.63) - open/close 45807 (time 1769081591.71 total 576.47 last 74.20) - open/close 46612 (time 1769081601.72 total 586.48 last 80.40) - open/close 47448 (time 1769081611.73 total 596.49 last 83.57) - open/close 48349 (time 1769081621.73 total 606.49 last 90.07) - open/close 49367 (time 1769081631.73 total 616.49 last 101.80) - open/close 50000 (time 1769081637.20 total 621.96 last 115.78) - open/close 51043 (time 1769081647.20 total 631.96 last 104.28) - open/close 52131 (time 1769081657.20 total 641.96 last 108.76) - open/close 53162 (time 1769081667.21 total 651.97 last 103.08) - open/close 54286 (time 1769081677.21 total 661.97 last 112.35) - open/close 55455 (time 1769081687.21 total 671.97 last 116.84) - open/close 56494 (time 1769081697.22 total 681.98 last 103.85) - open/close 57564 (time 1769081707.23 total 691.99 last 106.92) - open/close 58668 (time 1769081717.24 total 702.00 last 110.28) - open/close 59739 (time 1769081727.24 total 712.00 last 107.08) - open/close 60000 (time 1769081729.62 total 714.38 last 109.42) - open/close 61316 (time 1769081739.63 total 724.39 last 131.50) - open/close 62331 (time 1769081749.64 total 734.40 last 101.43) - open/close 63410 (time 1769081759.65 total 744.41 last 107.81) - open/close 64714 (time 1769081769.66 total 754.41 last 130.29) total: 64768 open/close in 754.85 seconds: 85.80 ops/second - /unlink 1256 (time 1769081782.29 total 10.01 last 125.52) - /unlink 2395 (time 1769081792.30 total 20.02 last 113.75) - /unlink 3534 (time 1769081802.31 total 30.02 last 113.87) - /unlink 4771 (time 1769081812.31 total 40.02 last 123.70) - /unlink 5854 (time 1769081822.31 total 50.03 last 108.27) - /unlink 6927 (time 1769081832.32 total 60.03 last 107.23) - /unlink 8040 (time 1769081842.32 total 70.04 last 111.24) - /unlink 9257 (time 1769081852.32 total 80.04 last 121.70) - /unlink 10000 (time 1769081858.79 total 86.50 last 114.93) - /unlink 11285 (time 1769081868.79 total 96.50 last 128.50) - /unlink 12642 (time 1769081878.79 total 106.50 last 135.70) - /unlink 14020 (time 1769081888.80 total 116.51 last 137.68) - /unlink 15256 (time 1769081898.80 total 126.51 last 123.56) - /unlink 16640 (time 1769081908.80 total 136.51 last 138.40) - /unlink 17859 (time 1769081918.80 total 146.52 last 121.89) - /unlink 19012 (time 1769081928.81 total 156.52 last 115.20) - /unlink 20000 (time 1769081936.58 total 164.29 last 127.19) - /unlink 21201 (time 1769081946.58 total 174.29 last 120.06) - /unlink 22470 (time 1769081956.59 total 184.30 last 126.82) - /unlink 23864 (time 1769081966.60 total 194.32 last 139.16) - /unlink 25806 (time 1769081976.60 total 204.32 last 194.16) - /unlink 27331 (time 1769081986.61 total 214.33 last 152.39) - /unlink 28970 (time 1769081996.61 total 224.33 last 163.87) - /unlink 30000 (time 1769082002.37 total 230.09 last 178.85) - /unlink 33205 (time 1769082012.37 total 240.09 last 320.44) - /unlink 36433 (time 1769082022.38 total 250.09 last 322.73) - /unlink 39977 (time 1769082032.38 total 260.09 last 354.40) - /unlink 43515 (time 1769082042.38 total 270.09 last 353.77) - /unlink 46498 (time 1769082052.38 total 280.09 last 298.30) - /unlink 50000 (time 1769082059.44 total 287.15 last 496.11) - /unlink 54389 (time 1769082069.44 total 297.15 last 438.89) - /unlink 58575 (time 1769082079.44 total 307.15 last 418.55) - /unlink 60000 (time 1769082082.87 total 310.59 last 414.57) total: 64768 /unlink in 319.60 seconds: 202.65 ops/second Waiting for MDT destroys to complete oleg140-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 - open/close 1658 (time 1769082114.15 total 10.00 last 165.75) - open/close 3279 (time 1769082124.15 total 20.00 last 162.07) - open/close 5117 (time 1769082134.15 total 30.01 last 183.76) - open/close 7063 (time 1769082144.16 total 40.01 last 194.59) - open/close 8736 (time 1769082154.16 total 50.01 last 167.21) - open/close 10000 (time 1769082162.35 total 58.20 last 154.40) - open/close 11508 (time 1769082172.35 total 68.20 last 150.78) - open/close 13153 (time 1769082182.35 total 78.20 last 164.46) - open/close 14600 (time 1769082192.35 total 88.20 last 144.69) - open/close 16202 (time 1769082202.35 total 98.21 last 160.15) - open/close 17811 (time 1769082212.36 total 108.22 last 160.75) - open/close 19276 (time 1769082222.37 total 118.22 last 146.48) - open/close 20000 (time 1769082228.16 total 124.01 last 124.90) - open/close 21455 (time 1769082238.16 total 134.02 last 145.49) - open/close 22973 (time 1769082248.17 total 144.02 last 151.71) - open/close 24562 (time 1769082258.17 total 154.02 last 158.90) - open/close 26069 (time 1769082268.18 total 164.03 last 150.59) - open/close 27801 (time 1769082278.18 total 174.03 last 173.19) - open/close 29959 (time 1769082288.18 total 184.03 last 215.77) - open/close 31817 (time 1769082298.19 total 194.04 last 185.68) - open/close 33391 (time 1769082308.19 total 204.04 last 157.34) - open/close 34853 (time 1769082318.19 total 214.04 last 146.18) - open/close 36743 (time 1769082328.19 total 224.05 last 188.96) - open/close 38711 (time 1769082338.19 total 234.05 last 196.79) - open/close 40000 (time 1769082345.56 total 241.41 last 175.09) - open/close 41516 (time 1769082355.56 total 251.41 last 151.58) - open/close 43113 (time 1769082365.56 total 261.41 last 159.65) - open/close 44648 (time 1769082375.56 total 271.41 last 153.48) - open/close 46237 (time 1769082385.57 total 281.42 last 158.82) - open/close 48330 (time 1769082395.57 total 291.42 last 209.28) - open/close 50000 (time 1769082403.28 total 299.13 last 216.50) - open/close 51946 (time 1769082413.28 total 309.13 last 194.58) - open/close 53950 (time 1769082423.28 total 319.14 last 200.38) - open/close 55702 (time 1769082433.29 total 329.14 last 175.17) - open/close 57747 (time 1769082443.29 total 339.14 last 204.44) - open/close 59791 (time 1769082453.29 total 349.14 last 204.33) - open/close 61443 (time 1769082463.30 total 359.15 last 165.13) - open/close 63057 (time 1769082473.30 total 369.15 last 161.38) - open/close 64729 (time 1769082483.30 total 379.15 last 167.14) total: 64768 open/close in 379.37 seconds: 170.72 ops/second - /unlink 2571 (time 1769082495.35 total 10.00 last 257.07) - /unlink 4917 (time 1769082505.36 total 20.00 last 234.53) - /unlink 7127 (time 1769082515.36 total 30.01 last 220.97) - /unlink 9585 (time 1769082525.36 total 40.01 last 245.77) - /unlink 11982 (time 1769082535.36 total 50.01 last 239.70) - /unlink 14611 (time 1769082545.37 total 60.01 last 262.69) - /unlink 17832 (time 1769082555.37 total 70.02 last 322.07) - /unlink 20000 (time 1769082562.49 total 77.14 last 304.46) - /unlink 23008 (time 1769082572.49 total 87.14 last 300.80) - /unlink 25846 (time 1769082582.49 total 97.14 last 283.76) - /unlink 28827 (time 1769082592.50 total 107.14 last 297.93) - /unlink 30000 (time 1769082595.74 total 110.39 last 361.69) - /unlink 32815 (time 1769082605.74 total 120.39 last 281.48) - /unlink 36343 (time 1769082615.74 total 130.39 last 352.78) - /unlink 40000 (time 1769082625.62 total 140.27 last 370.25) - /unlink 43976 (time 1769082635.62 total 150.27 last 397.49) - /unlink 47915 (time 1769082645.62 total 160.27 last 393.86) - /unlink 50000 (time 1769082651.22 total 165.87 last 372.44) - /unlink 53331 (time 1769082661.23 total 175.87 last 332.90) - /unlink 56147 (time 1769082671.23 total 185.87 last 281.59) - /unlink 60000 (time 1769082680.74 total 195.39 last 404.97) - /unlink 64637 (time 1769082690.74 total 205.39 last 463.68) total: 64768 /unlink in 205.70 seconds: 314.87 ops/second Waiting for MDT destroys to complete oleg140-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 - open/close 2832 (time 1769082713.02 total 10.00 last 283.18) - open/close 5549 (time 1769082723.02 total 20.00 last 271.66) - open/close 7669 (time 1769082733.02 total 30.00 last 211.95) - open/close 9993 (time 1769082743.02 total 40.01 last 232.38) - open/close 11998 (time 1769082753.02 total 50.01 last 200.47) - open/close 14497 (time 1769082763.02 total 60.01 last 249.88) - open/close 17152 (time 1769082773.03 total 70.01 last 265.38) - open/close 20000 (time 1769082781.79 total 78.78 last 324.93) - open/close 23164 (time 1769082791.80 total 88.78 last 316.36) - open/close 26642 (time 1769082801.80 total 98.78 last 347.75) - open/close 29876 (time 1769082811.80 total 108.78 last 323.37) - open/close 33078 (time 1769082821.80 total 118.79 last 320.08) - open/close 35954 (time 1769082831.80 total 128.79 last 287.54) - open/close 38588 (time 1769082841.80 total 138.79 last 263.38) - open/close 40000 (time 1769082845.61 total 142.59 last 371.13) - open/close 43741 (time 1769082855.61 total 152.59 last 374.10) - open/close 46917 (time 1769082865.61 total 162.60 last 317.51) - open/close 49275 (time 1769082875.61 total 172.60 last 235.76) - open/close 50000 (time 1769082878.15 total 175.13 last 286.15) - open/close 53201 (time 1769082888.15 total 185.13 last 320.09) - open/close 56735 (time 1769082898.15 total 195.13 last 353.38) - open/close 60000 (time 1769082906.84 total 203.83 last 375.45) total: 64768 open/close in 213.77 seconds: 302.97 ops/second - /unlink 7331 (time 1769082928.13 total 10.00 last 733.00) - /unlink 10000 (time 1769082931.94 total 13.82 last 699.70) - /unlink 15646 (time 1769082941.94 total 23.82 last 564.54) - /unlink 20000 (time 1769082947.67 total 29.55 last 759.83) - /unlink 26688 (time 1769082957.67 total 39.55 last 668.80) - /unlink 30000 (time 1769082962.05 total 43.93 last 756.18) - /unlink 37096 (time 1769082972.05 total 53.93 last 709.54) - /unlink 40000 (time 1769082975.42 total 57.29 last 863.08) - /unlink 47315 (time 1769082985.42 total 67.29 last 731.44) - /unlink 50000 (time 1769082988.92 total 70.79 last 767.94) - /unlink 55587 (time 1769082998.92 total 80.79 last 558.58) - /unlink 58331 (time 1769083008.92 total 90.80 last 274.32) - /unlink 60000 (time 1769083016.97 total 98.85 last 207.27) - /unlink 61854 (time 1769083026.97 total 108.85 last 185.40) - /unlink 63941 (time 1769083036.97 total 118.85 last 208.67) total: 64768 /unlink in 122.47 seconds: 528.84 ops/second Waiting for MDT destroys to complete oleg140-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 - open/close 1548 (time 1769083060.63 total 10.00 last 154.78) - open/close 3170 (time 1769083070.63 total 20.01 last 162.10) - open/close 4830 (time 1769083080.63 total 30.01 last 165.98) - open/close 6518 (time 1769083090.64 total 40.01 last 168.77) - open/close 8387 (time 1769083100.64 total 50.01 last 186.83) - open/close 10000 (time 1769083109.25 total 58.62 last 187.41) - open/close 11829 (time 1769083119.25 total 68.62 last 182.87) - open/close 14069 (time 1769083129.25 total 78.63 last 223.92) - open/close 16106 (time 1769083139.25 total 88.63 last 203.65) - open/close 17592 (time 1769083149.26 total 98.64 last 148.48) - open/close 19577 (time 1769083159.26 total 108.64 last 198.49) - open/close 20000 (time 1769083161.64 total 111.02 last 177.63) - open/close 21835 (time 1769083171.65 total 121.02 last 183.49) - open/close 24353 (time 1769083181.65 total 131.02 last 251.77) - open/close 26442 (time 1769083191.65 total 141.02 last 208.81) - open/close 28389 (time 1769083201.66 total 151.03 last 194.61) - open/close 30000 (time 1769083210.78 total 160.15 last 176.62) - open/close 31942 (time 1769083220.78 total 170.15 last 194.17) - open/close 34447 (time 1769083230.78 total 180.15 last 250.49) - open/close 36642 (time 1769083240.78 total 190.15 last 219.44) - open/close 37836 (time 1769083250.78 total 200.16 last 119.36) - open/close 38818 (time 1769083260.96 total 210.33 last 96.54) - open/close 39853 (time 1769083270.96 total 220.34 last 103.42) - open/close 40871 (time 1769083280.97 total 230.34 last 101.76) - open/close 41981 (time 1769083290.97 total 240.34 last 110.99) - open/close 43326 (time 1769083300.98 total 250.35 last 134.40) - open/close 44712 (time 1769083310.98 total 260.35 last 138.54) - open/close 45983 (time 1769083320.99 total 270.36 last 127.04) - open/close 47196 (time 1769083330.99 total 280.36 last 121.28) - open/close 48417 (time 1769083340.99 total 290.36 last 122.09) - open/close 50000 (time 1769083350.21 total 299.59 last 171.60) - open/close 51692 (time 1769083360.22 total 309.59 last 169.12) - open/close 53431 (time 1769083370.22 total 319.60 last 173.80) - open/close 54993 (time 1769083380.23 total 329.60 last 156.16) - open/close 56456 (time 1769083390.23 total 339.60 last 146.25) - open/close 57848 (time 1769083400.24 total 349.62 last 139.02) - open/close 59120 (time 1769083410.25 total 359.62 last 127.15) - open/close 60000 (time 1769083415.66 total 365.03 last 162.62) - open/close 61598 (time 1769083425.66 total 375.03 last 159.77) - open/close 63332 (time 1769083435.66 total 385.04 last 173.31) total: 64768 open/close in 394.43 seconds: 164.20 ops/second - /unlink 3658 (time 1769083457.06 total 10.00 last 365.78) - /unlink 7314 (time 1769083467.07 total 20.00 last 365.50) - /unlink 10000 (time 1769083474.94 total 27.88 last 340.99) - /unlink 14181 (time 1769083484.94 total 37.88 last 418.04) - /unlink 19123 (time 1769083494.95 total 47.88 last 494.14) - /unlink 20000 (time 1769083497.16 total 50.10 last 395.83) - /unlink 23457 (time 1769083507.16 total 60.10 last 345.66) - /unlink 26845 (time 1769083517.16 total 70.10 last 338.74) - /unlink 30000 (time 1769083524.75 total 77.68 last 416.18) - /unlink 34833 (time 1769083534.75 total 87.68 last 483.28) - /unlink 38704 (time 1769083544.75 total 97.68 last 387.03) - /unlink 40000 (time 1769083548.12 total 101.06 last 383.80) - /unlink 44769 (time 1769083558.13 total 111.06 last 476.82) - /unlink 48426 (time 1769083568.13 total 121.07 last 365.61) - /unlink 50000 (time 1769083571.26 total 124.20 last 502.39) - /unlink 52054 (time 1769083581.26 total 134.20 last 205.39) - /unlink 53264 (time 1769083591.27 total 144.20 last 120.95) - /unlink 54518 (time 1769083601.27 total 154.20 last 125.39) - /unlink 55728 (time 1769083611.27 total 164.21 last 120.93) - /unlink 56709 (time 1769083621.28 total 174.21 last 98.06) - /unlink 57776 (time 1769083631.28 total 184.22 last 106.66) - /unlink 59171 (time 1769083641.31 total 194.25 last 139.04) - /unlink 60000 (time 1769083647.31 total 200.25 last 138.14) - /unlink 61532 (time 1769083657.32 total 210.26 last 153.11) - /unlink 63130 (time 1769083667.32 total 220.26 last 159.72) total: 64768 /unlink in 229.72 seconds: 281.95 ops/second Waiting for MDT destroys to complete oleg140-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg140-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2