== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:42:36 (1753742556) Stopping clients: oleg336-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg336-client.virtnet /mnt/lustre2 (opts:-f) oleg336-server: oleg336-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg336-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg336-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg336-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg336-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 840 (time 1753742654.14 total 10.01 last 83.93) - open/close 1613 (time 1753742664.16 total 20.02 last 77.20) - open/close 2358 (time 1753742674.16 total 30.03 last 74.46) - open/close 3132 (time 1753742684.17 total 40.03 last 77.37) - open/close 3912 (time 1753742694.17 total 50.04 last 77.95) - open/close 4706 (time 1753742704.18 total 60.05 last 79.30) - open/close 5489 (time 1753742714.19 total 70.05 last 78.26) - open/close 6344 (time 1753742724.20 total 80.06 last 85.42) - open/close 7052 (time 1753742734.21 total 90.07 last 70.74) - open/close 7823 (time 1753742744.21 total 100.07 last 77.09) - open/close 8616 (time 1753742754.21 total 110.07 last 79.29) - open/close 9429 (time 1753742764.21 total 120.08 last 81.25) - open/close 10000 (time 1753742771.59 total 127.46 last 77.42) - open/close 10742 (time 1753742781.59 total 137.46 last 74.17) - open/close 11399 (time 1753742791.61 total 147.47 last 65.60) - open/close 12097 (time 1753742801.61 total 157.48 last 69.77) - open/close 12874 (time 1753742811.63 total 167.49 last 77.60) - open/close 13672 (time 1753742821.63 total 177.50 last 79.75) - open/close 14402 (time 1753742831.64 total 187.51 last 72.95) - open/close 15227 (time 1753742841.64 total 197.51 last 82.50) - open/close 16005 (time 1753742851.66 total 207.52 last 77.66) - open/close 16760 (time 1753742861.67 total 217.53 last 75.44) - open/close 17501 (time 1753742871.67 total 227.53 last 74.09) - open/close 18191 (time 1753742881.68 total 237.54 last 68.93) - open/close 18896 (time 1753742891.69 total 247.56 last 70.41) - open/close 19604 (time 1753742901.72 total 257.58 last 70.60) - open/close 20000 (time 1753742906.85 total 262.71 last 77.21) - open/close 20773 (time 1753742916.85 total 272.72 last 77.26) - open/close 21605 (time 1753742926.85 total 282.72 last 83.18) - open/close 22438 (time 1753742936.85 total 292.72 last 83.30) - open/close 23238 (time 1753742946.86 total 302.73 last 79.93) - open/close 24040 (time 1753742956.87 total 312.74 last 80.13) - open/close 24823 (time 1753742966.89 total 322.75 last 78.18) - open/close 25610 (time 1753742976.90 total 332.76 last 78.62) - open/close 26403 (time 1753742986.90 total 342.77 last 79.24) - open/close 27044 (time 1753742996.91 total 352.78 last 64.03) - open/close 27696 (time 1753743006.92 total 362.78 last 65.19) - open/close 28409 (time 1753743016.93 total 372.80 last 71.20) - open/close 29155 (time 1753743026.94 total 382.81 last 74.53) - open/close 29938 (time 1753743036.95 total 392.81 last 78.23) - open/close 30742 (time 1753743046.95 total 402.82 last 80.38) - open/close 31508 (time 1753743056.95 total 412.82 last 76.59) - open/close 32279 (time 1753743066.97 total 422.83 last 77.00) - open/close 32994 (time 1753743076.97 total 432.84 last 71.48) - open/close 33700 (time 1753743086.98 total 442.84 last 70.53) - open/close 34405 (time 1753743096.98 total 452.84 last 70.50) - open/close 35155 (time 1753743106.99 total 462.85 last 74.95) - open/close 35846 (time 1753743117.00 total 472.87 last 68.98) - open/close 36600 (time 1753743127.00 total 482.87 last 75.40) - open/close 37329 (time 1753743137.02 total 492.89 last 72.76) - open/close 38025 (time 1753743147.03 total 502.89 last 69.57) - open/close 38792 (time 1753743157.04 total 512.91 last 76.60) - open/close 39487 (time 1753743167.05 total 522.91 last 69.44) - open/close 40000 (time 1753743173.81 total 529.68 last 75.85) - open/close 40798 (time 1753743183.81 total 539.68 last 79.78) - open/close 41571 (time 1753743193.83 total 549.69 last 77.18) - open/close 42332 (time 1753743203.83 total 559.70 last 76.06) - open/close 43109 (time 1753743213.84 total 569.71 last 77.62) - open/close 43843 (time 1753743223.86 total 579.72 last 73.32) - open/close 44548 (time 1753743233.87 total 589.73 last 70.43) - open/close 45068 (time 1753743243.88 total 599.75 last 51.92) - open/close 45450 (time 1753743253.90 total 609.77 last 38.11) - open/close 45910 (time 1753743263.92 total 619.79 last 45.93) - open/close 46422 (time 1753743273.94 total 629.81 last 51.07) - open/close 47195 (time 1753743283.95 total 639.81 last 77.28) - open/close 48005 (time 1753743293.95 total 649.81 last 80.99) - open/close 48760 (time 1753743303.95 total 659.81 last 75.49) - open/close 49521 (time 1753743313.97 total 669.83 last 75.95) - open/close 50000 (time 1753743319.75 total 675.62 last 82.83) - open/close 50858 (time 1753743329.75 total 685.62 last 85.77) - open/close 51568 (time 1753743339.75 total 695.62 last 71.00) - open/close 52182 (time 1753743349.77 total 705.64 last 61.30) - open/close 52762 (time 1753743359.79 total 715.66 last 57.88) - open/close 53416 (time 1753743369.80 total 725.66 last 65.38) - open/close 54028 (time 1753743379.80 total 735.66 last 61.18) - open/close 54700 (time 1753743389.81 total 745.68 last 67.12) - open/close 55456 (time 1753743399.81 total 755.68 last 75.59) - open/close 56183 (time 1753743409.82 total 765.69 last 72.62) - open/close 56845 (time 1753743419.84 total 775.70 last 66.11) - open/close 57571 (time 1753743429.84 total 785.70 last 72.60) - open/close 58301 (time 1753743439.84 total 795.71 last 72.96) - open/close 59043 (time 1753743449.84 total 805.71 last 74.19) - open/close 59771 (time 1753743459.86 total 815.72 last 72.71) - open/close 60000 (time 1753743462.99 total 818.86 last 73.04) - open/close 60770 (time 1753743473.00 total 828.87 last 76.93) - open/close 61593 (time 1753743483.01 total 838.87 last 82.24) - open/close 62401 (time 1753743493.01 total 848.88 last 80.76) - open/close 63103 (time 1753743503.02 total 858.89 last 70.12) - open/close 63835 (time 1753743513.03 total 868.89 last 73.17) - open/close 64618 (time 1753743523.03 total 878.90 last 78.29) total: 64768 open/close in 880.60 seconds: 73.55 ops/second - /unlink 741 (time 1753743537.75 total 10.00 last 74.09) - /unlink 1471 (time 1753743547.77 total 20.02 last 72.85) - /unlink 2213 (time 1753743557.79 total 30.04 last 74.08) - /unlink 2978 (time 1753743567.80 total 40.05 last 76.40) - /unlink 3749 (time 1753743577.81 total 50.06 last 77.03) - /unlink 4488 (time 1753743587.82 total 60.07 last 73.83) - /unlink 5227 (time 1753743597.83 total 70.08 last 73.80) - /unlink 5984 (time 1753743607.85 total 80.10 last 75.59) - /unlink 6771 (time 1753743617.86 total 90.11 last 78.60) - /unlink 7610 (time 1753743627.86 total 100.11 last 83.88) - /unlink 8520 (time 1753743637.86 total 110.11 last 90.99) - /unlink 9356 (time 1753743647.87 total 120.12 last 83.55) - /unlink 10000 (time 1753743656.68 total 128.94 last 73.05) - /unlink 10668 (time 1753743666.69 total 138.94 last 66.77) - /unlink 11383 (time 1753743676.69 total 148.94 last 71.49) - /unlink 12023 (time 1753743686.69 total 158.94 last 63.99) - /unlink 12609 (time 1753743696.70 total 168.95 last 58.54) - /unlink 13293 (time 1753743706.72 total 178.97 last 68.27) - /unlink 13969 (time 1753743716.72 total 188.97 last 67.59) - /unlink 14756 (time 1753743726.74 total 198.99 last 78.57) - /unlink 15495 (time 1753743736.75 total 209.01 last 73.78) - /unlink 16205 (time 1753743746.76 total 219.01 last 71.00) - /unlink 16972 (time 1753743756.77 total 229.02 last 76.59) - /unlink 17712 (time 1753743766.78 total 239.03 last 73.95) - /unlink 18453 (time 1753743776.78 total 249.03 last 74.06) - /unlink 19131 (time 1753743786.79 total 259.04 last 67.76) - /unlink 19871 (time 1753743796.80 total 269.05 last 73.90) - /unlink 20573 (time 1753743806.81 total 279.07 last 70.12) - /unlink 21326 (time 1753743816.81 total 289.07 last 75.30) - /unlink 22118 (time 1753743826.82 total 299.07 last 79.14) - /unlink 22830 (time 1753743836.83 total 309.08 last 71.15) - /unlink 23557 (time 1753743846.83 total 319.08 last 72.69) - /unlink 24206 (time 1753743856.83 total 329.08 last 64.90) - /unlink 24861 (time 1753743866.85 total 339.10 last 65.39) - /unlink 25591 (time 1753743876.86 total 349.11 last 72.93) - /unlink 26379 (time 1753743886.87 total 359.12 last 78.70) - /unlink 27115 (time 1753743896.87 total 369.13 last 73.57) - /unlink 27935 (time 1753743906.88 total 379.13 last 81.98) - /unlink 28759 (time 1753743916.88 total 389.13 last 82.35) - /unlink 29587 (time 1753743926.88 total 399.13 last 82.79) - /unlink 30000 (time 1753743931.90 total 404.15 last 82.28) - /unlink 30838 (time 1753743941.91 total 414.16 last 83.75) - /unlink 31639 (time 1753743951.92 total 424.17 last 79.99) - /unlink 32412 (time 1753743961.92 total 434.18 last 77.28) - /unlink 33218 (time 1753743971.93 total 444.18 last 80.56) - /unlink 34084 (time 1753743981.96 total 454.21 last 86.34) - /unlink 34803 (time 1753743991.97 total 464.22 last 71.81) - /unlink 35472 (time 1753744001.99 total 474.24 last 66.76) - /unlink 36299 (time 1753744012.03 total 484.28 last 82.40) - /unlink 37113 (time 1753744022.06 total 494.32 last 81.11) - /unlink 37915 (time 1753744032.07 total 504.32 last 80.15) - /unlink 38702 (time 1753744042.08 total 514.33 last 78.66) - /unlink 39480 (time 1753744052.08 total 524.33 last 77.77) - /unlink 40000 (time 1753744058.96 total 531.21 last 75.57) - /unlink 40790 (time 1753744068.97 total 541.22 last 78.92) - /unlink 41593 (time 1753744078.97 total 551.22 last 80.29) - /unlink 42330 (time 1753744088.98 total 561.23 last 73.63) - /unlink 43086 (time 1753744098.98 total 571.23 last 75.59) - /unlink 43899 (time 1753744108.98 total 581.24 last 81.28) - /unlink 44661 (time 1753744118.99 total 591.25 last 76.12) - /unlink 45445 (time 1753744129.00 total 601.25 last 78.33) - /unlink 46232 (time 1753744139.01 total 611.26 last 78.64) - /unlink 47026 (time 1753744149.01 total 621.27 last 79.36) - /unlink 47882 (time 1753744159.02 total 631.28 last 85.53) - /unlink 48700 (time 1753744169.04 total 641.29 last 81.66) - /unlink 49530 (time 1753744179.05 total 651.30 last 82.95) - /unlink 50000 (time 1753744184.92 total 657.18 last 79.95) - /unlink 50774 (time 1753744194.93 total 667.18 last 77.38) - /unlink 51570 (time 1753744204.93 total 677.19 last 79.55) - /unlink 52315 (time 1753744214.94 total 687.20 last 74.43) - /unlink 53088 (time 1753744224.94 total 697.20 last 77.29) - /unlink 53879 (time 1753744234.97 total 707.22 last 78.93) - /unlink 54674 (time 1753744244.97 total 717.22 last 79.49) - /unlink 55450 (time 1753744254.97 total 727.23 last 77.54) - /unlink 56232 (time 1753744264.98 total 737.24 last 78.13) - /unlink 57017 (time 1753744274.99 total 747.24 last 78.49) - /unlink 57833 (time 1753744284.99 total 757.24 last 81.59) - /unlink 58657 (time 1753744295.00 total 767.25 last 82.31) - /unlink 59438 (time 1753744305.01 total 777.26 last 77.99) - /unlink 60000 (time 1753744311.86 total 784.12 last 82.01) - /unlink 60789 (time 1753744321.87 total 794.12 last 78.90) - /unlink 61593 (time 1753744331.87 total 804.12 last 80.38) - /unlink 62411 (time 1753744341.88 total 814.13 last 81.74) - /unlink 63245 (time 1753744351.89 total 824.14 last 83.32) - /unlink 64046 (time 1753744361.89 total 834.14 last 80.09) total: 64768 /unlink in 843.83 seconds: 76.75 ops/second Waiting for MDT destroys to complete oleg336-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 - open/close 772 (time 1753744396.69 total 10.00 last 77.20) - open/close 1562 (time 1753744406.71 total 20.02 last 78.85) - open/close 2355 (time 1753744416.74 total 30.05 last 79.10) - open/close 3191 (time 1753744426.74 total 40.05 last 83.59) - open/close 4035 (time 1753744436.74 total 50.05 last 84.37) - open/close 4865 (time 1753744446.74 total 60.05 last 82.99) - open/close 5709 (time 1753744456.75 total 70.06 last 84.33) - open/close 6575 (time 1753744466.75 total 80.06 last 86.59) - open/close 7239 (time 1753744476.78 total 90.09 last 66.22) - open/close 7852 (time 1753744486.79 total 100.10 last 61.25) - open/close 8531 (time 1753744496.79 total 110.10 last 67.89) - open/close 9209 (time 1753744506.81 total 120.12 last 67.67) - open/close 9946 (time 1753744516.82 total 130.13 last 73.63) - open/close 10832 (time 1753744526.83 total 140.14 last 88.45) - open/close 11609 (time 1753744536.84 total 150.15 last 77.67) - open/close 12382 (time 1753744546.85 total 160.16 last 77.20) - open/close 13157 (time 1753744556.85 total 170.16 last 77.47) - open/close 13969 (time 1753744566.86 total 180.17 last 81.13) - open/close 14848 (time 1753744576.87 total 190.18 last 87.86) - open/close 15686 (time 1753744586.87 total 200.18 last 83.79) - open/close 16543 (time 1753744596.88 total 210.18 last 85.63) - open/close 17344 (time 1753744606.88 total 220.19 last 80.08) - open/close 18096 (time 1753744616.88 total 230.19 last 75.18) - open/close 18923 (time 1753744626.89 total 240.20 last 82.62) - open/close 19688 (time 1753744636.89 total 250.20 last 76.48) - open/close 20000 (time 1753744641.09 total 254.40 last 74.35) - open/close 20809 (time 1753744651.09 total 264.40 last 80.88) - open/close 21643 (time 1753744661.09 total 274.40 last 83.39) - open/close 22462 (time 1753744671.09 total 284.40 last 81.89) - open/close 23241 (time 1753744681.11 total 294.42 last 77.80) - open/close 23998 (time 1753744691.12 total 304.43 last 75.64) - open/close 24721 (time 1753744701.12 total 314.43 last 72.26) - open/close 25490 (time 1753744711.12 total 324.43 last 76.88) - open/close 26222 (time 1753744721.13 total 334.44 last 73.18) - open/close 26975 (time 1753744731.16 total 344.47 last 75.04) - open/close 27817 (time 1753744741.17 total 354.48 last 84.11) - open/close 28670 (time 1753744751.17 total 364.48 last 85.29) - open/close 29509 (time 1753744761.18 total 374.49 last 83.87) - open/close 30000 (time 1753744767.13 total 380.44 last 82.42) - open/close 30936 (time 1753744777.14 total 390.45 last 93.57) - open/close 31782 (time 1753744787.14 total 400.45 last 84.57) - open/close 32625 (time 1753744797.16 total 410.46 last 84.18) - open/close 33435 (time 1753744807.16 total 420.47 last 80.94) - open/close 34255 (time 1753744817.17 total 430.48 last 81.93) - open/close 35054 (time 1753744827.19 total 440.50 last 79.78) - open/close 35877 (time 1753744837.19 total 450.50 last 82.23) - open/close 36684 (time 1753744847.21 total 460.52 last 80.60) - open/close 37471 (time 1753744857.21 total 470.52 last 78.69) - open/close 38165 (time 1753744867.22 total 480.53 last 69.34) - open/close 38903 (time 1753744877.23 total 490.54 last 73.70) - open/close 39638 (time 1753744887.24 total 500.55 last 73.42) - open/close 40000 (time 1753744893.69 total 506.99 last 56.17) - open/close 40598 (time 1753744903.69 total 517.00 last 59.75) - open/close 41310 (time 1753744913.69 total 527.00 last 71.20) - open/close 42098 (time 1753744923.70 total 537.01 last 78.76) - open/close 42931 (time 1753744933.70 total 547.01 last 83.26) - open/close 43805 (time 1753744943.71 total 557.02 last 87.37) - open/close 44654 (time 1753744953.71 total 567.02 last 84.83) - open/close 45482 (time 1753744963.72 total 577.03 last 82.78) - open/close 46307 (time 1753744973.72 total 587.03 last 82.45) - open/close 47185 (time 1753744983.73 total 597.04 last 87.73) - open/close 48039 (time 1753744993.74 total 607.05 last 85.31) - open/close 48833 (time 1753745003.74 total 617.05 last 79.39) - open/close 49582 (time 1753745013.75 total 627.06 last 74.87) - open/close 50000 (time 1753745018.67 total 631.98 last 84.86) - open/close 50794 (time 1753745028.69 total 642.00 last 79.29) - open/close 51628 (time 1753745038.69 total 652.00 last 83.35) - open/close 52415 (time 1753745048.71 total 662.01 last 78.60) - open/close 53177 (time 1753745058.71 total 672.02 last 76.16) - open/close 54039 (time 1753745068.72 total 682.02 last 86.16) - open/close 54944 (time 1753745078.72 total 692.03 last 90.44) - open/close 55784 (time 1753745088.74 total 702.05 last 83.84) - open/close 56577 (time 1753745098.75 total 712.06 last 79.24) - open/close 57423 (time 1753745108.76 total 722.07 last 84.54) - open/close 58206 (time 1753745118.76 total 732.07 last 78.27) - open/close 58945 (time 1753745128.76 total 742.07 last 73.89) - open/close 59780 (time 1753745138.76 total 752.07 last 83.48) - open/close 60000 (time 1753745141.18 total 754.49 last 91.02) - open/close 60810 (time 1753745151.19 total 764.50 last 80.91) - open/close 61493 (time 1753745161.22 total 774.52 last 68.14) - open/close 62186 (time 1753745171.22 total 784.53 last 69.29) - open/close 62954 (time 1753745181.24 total 794.55 last 76.62) - open/close 63732 (time 1753745191.25 total 804.55 last 77.76) - open/close 64462 (time 1753745201.26 total 814.56 last 72.93) total: 64768 open/close in 818.72 seconds: 79.11 ops/second - /unlink 800 (time 1753745218.96 total 10.01 last 79.93) - /unlink 1567 (time 1753745228.96 total 20.02 last 76.64) - /unlink 2259 (time 1753745238.96 total 30.02 last 69.19) - /unlink 2933 (time 1753745248.97 total 40.03 last 67.35) - /unlink 3601 (time 1753745258.98 total 50.03 last 66.75) - /unlink 4272 (time 1753745268.98 total 60.04 last 67.08) - /unlink 4991 (time 1753745278.99 total 70.04 last 71.85) - /unlink 5751 (time 1753745288.99 total 80.05 last 75.97) - /unlink 6542 (time 1753745299.00 total 90.06 last 79.01) - /unlink 7287 (time 1753745309.02 total 100.07 last 74.38) - /unlink 7987 (time 1753745319.02 total 110.07 last 70.00) - /unlink 8731 (time 1753745329.02 total 120.08 last 74.37) - /unlink 9471 (time 1753745339.02 total 130.08 last 74.00) - /unlink 10000 (time 1753745346.36 total 137.42 last 72.07) - /unlink 10659 (time 1753745356.37 total 147.43 last 65.85) - /unlink 11383 (time 1753745366.37 total 157.43 last 72.38) - /unlink 12185 (time 1753745376.40 total 167.45 last 80.00) - /unlink 12955 (time 1753745386.41 total 177.46 last 76.96) - /unlink 13691 (time 1753745396.41 total 187.46 last 73.56) - /unlink 14362 (time 1753745406.42 total 197.47 last 67.05) - /unlink 15079 (time 1753745416.42 total 207.48 last 71.66) - /unlink 15808 (time 1753745426.43 total 217.48 last 72.89) - /unlink 16551 (time 1753745436.43 total 227.48 last 74.29) - /unlink 17271 (time 1753745446.43 total 237.48 last 71.98) - /unlink 18049 (time 1753745456.44 total 247.49 last 77.71) - /unlink 18777 (time 1753745466.45 total 257.50 last 72.74) - /unlink 19584 (time 1753745476.46 total 267.51 last 80.62) - /unlink 20000 (time 1753745481.96 total 273.01 last 75.66) - /unlink 20730 (time 1753745491.97 total 283.03 last 72.88) - /unlink 21497 (time 1753745501.98 total 293.04 last 76.62) - /unlink 22269 (time 1753745511.99 total 303.05 last 77.11) - /unlink 23074 (time 1753745522.00 total 313.05 last 80.47) - /unlink 23679 (time 1753745532.01 total 323.06 last 60.42) - /unlink 24206 (time 1753745542.02 total 333.07 last 52.66) - /unlink 24769 (time 1753745552.04 total 343.09 last 56.18) - /unlink 25438 (time 1753745562.04 total 353.10 last 66.89) - /unlink 26150 (time 1753745572.05 total 363.10 last 71.17) - /unlink 26920 (time 1753745582.05 total 373.10 last 76.98) - /unlink 27625 (time 1753745592.06 total 383.11 last 70.44) - /unlink 28262 (time 1753745602.06 total 393.12 last 63.66) - /unlink 28881 (time 1753745612.07 total 403.13 last 61.83) - /unlink 29555 (time 1753745622.08 total 413.14 last 67.35) - /unlink 30000 (time 1753745629.09 total 420.14 last 63.54) - /unlink 30705 (time 1753745639.10 total 430.16 last 70.39) - /unlink 31442 (time 1753745649.11 total 440.17 last 73.61) - /unlink 32163 (time 1753745659.12 total 450.17 last 72.06) - /unlink 32938 (time 1753745669.13 total 460.18 last 77.46) - /unlink 33658 (time 1753745679.13 total 470.18 last 71.97) - /unlink 34407 (time 1753745689.13 total 480.18 last 74.89) - /unlink 35188 (time 1753745699.14 total 490.19 last 78.06) - /unlink 35916 (time 1753745709.15 total 500.20 last 72.73) - /unlink 36685 (time 1753745719.15 total 510.21 last 76.84) - /unlink 37450 (time 1753745729.16 total 520.21 last 76.45) - /unlink 38207 (time 1753745739.16 total 530.22 last 75.67) - /unlink 39017 (time 1753745749.17 total 540.22 last 80.94) - /unlink 39788 (time 1753745759.17 total 550.23 last 77.08) - /unlink 40000 (time 1753745761.76 total 552.81 last 82.05) - /unlink 40836 (time 1753745771.76 total 562.82 last 83.55) - /unlink 41647 (time 1753745781.77 total 572.83 last 81.01) - /unlink 42394 (time 1753745791.78 total 582.84 last 74.63) - /unlink 43240 (time 1753745801.79 total 592.84 last 84.57) - /unlink 44067 (time 1753745811.79 total 602.84 last 82.66) - /unlink 44839 (time 1753745821.79 total 612.84 last 77.20) - /unlink 45658 (time 1753745831.79 total 622.85 last 81.87) - /unlink 46423 (time 1753745841.80 total 632.85 last 76.48) - /unlink 47210 (time 1753745851.80 total 642.85 last 78.70) - /unlink 48003 (time 1753745861.80 total 652.85 last 79.27) - /unlink 48766 (time 1753745871.81 total 662.86 last 76.24) - /unlink 49551 (time 1753745881.81 total 672.86 last 78.48) - /unlink 50000 (time 1753745887.51 total 678.56 last 78.81) - /unlink 50799 (time 1753745897.51 total 688.57 last 79.87) - /unlink 51612 (time 1753745907.52 total 698.57 last 81.25) - /unlink 52444 (time 1753745917.52 total 708.57 last 83.19) - /unlink 53321 (time 1753745927.52 total 718.57 last 87.68) - /unlink 54211 (time 1753745937.53 total 728.58 last 88.96) - /unlink 55048 (time 1753745947.53 total 738.58 last 83.69) - /unlink 55956 (time 1753745957.53 total 748.58 last 90.79) - /unlink 56937 (time 1753745967.53 total 758.59 last 98.05) - /unlink 57829 (time 1753745977.54 total 768.59 last 89.18) - /unlink 58691 (time 1753745987.54 total 778.59 last 86.15) - /unlink 59619 (time 1753745997.54 total 788.60 last 92.78) - /unlink 60000 (time 1753746001.65 total 792.70 last 92.87) - /unlink 60868 (time 1753746011.66 total 802.71 last 86.68) - /unlink 61656 (time 1753746021.66 total 812.71 last 78.80) - /unlink 62511 (time 1753746031.66 total 822.72 last 85.47) - /unlink 63416 (time 1753746041.67 total 832.72 last 90.44) - /unlink 64234 (time 1753746051.67 total 842.72 last 81.80) total: 64768 /unlink in 849.52 seconds: 76.24 ops/second Waiting for MDT destroys to complete oleg336-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 - open/close 881 (time 1753746080.39 total 10.01 last 88.03) - open/close 1693 (time 1753746090.40 total 20.02 last 81.12) - open/close 2601 (time 1753746100.40 total 30.02 last 90.74) - open/close 3555 (time 1753746110.42 total 40.04 last 95.25) - open/close 4472 (time 1753746120.43 total 50.05 last 91.61) - open/close 5369 (time 1753746130.43 total 60.05 last 89.66) - open/close 6251 (time 1753746140.44 total 70.06 last 88.16) - open/close 7200 (time 1753746150.45 total 80.07 last 94.84) - open/close 8175 (time 1753746160.46 total 90.08 last 97.41) - open/close 9072 (time 1753746170.46 total 100.08 last 89.65) - open/close 10000 (time 1753746179.84 total 109.46 last 98.99) - open/close 10992 (time 1753746189.84 total 119.46 last 99.11) - open/close 11944 (time 1753746199.85 total 129.48 last 95.10) - open/close 12933 (time 1753746209.86 total 139.48 last 98.87) - open/close 13869 (time 1753746219.86 total 149.48 last 93.59) - open/close 14904 (time 1753746229.87 total 159.49 last 103.42) - open/close 15850 (time 1753746239.87 total 169.49 last 94.60) - open/close 16772 (time 1753746249.87 total 179.49 last 92.18) - open/close 17636 (time 1753746259.87 total 189.49 last 86.37) - open/close 18498 (time 1753746269.87 total 199.49 last 86.17) - open/close 19379 (time 1753746279.88 total 209.50 last 88.03) - open/close 20000 (time 1753746286.70 total 216.32 last 91.06) - open/close 20866 (time 1753746296.70 total 226.32 last 86.59) - open/close 21746 (time 1753746306.72 total 236.34 last 87.82) - open/close 22608 (time 1753746316.74 total 246.36 last 86.05) - open/close 23510 (time 1753746326.76 total 256.38 last 89.99) - open/close 24398 (time 1753746336.79 total 266.41 last 88.56) - open/close 25324 (time 1753746346.83 total 276.45 last 92.22) - open/close 26252 (time 1753746356.87 total 286.49 last 92.50) - open/close 27181 (time 1753746366.87 total 296.49 last 92.84) - open/close 28049 (time 1753746376.88 total 306.50 last 86.73) - open/close 28972 (time 1753746386.89 total 316.51 last 92.22) - open/close 29949 (time 1753746396.90 total 326.52 last 97.63) - open/close 30850 (time 1753746406.90 total 336.52 last 90.08) - open/close 31770 (time 1753746416.90 total 346.52 last 91.95) - open/close 32640 (time 1753746426.91 total 356.53 last 86.96) - open/close 33579 (time 1753746436.91 total 366.53 last 93.89) - open/close 34513 (time 1753746446.91 total 376.53 last 93.37) - open/close 35509 (time 1753746456.92 total 386.54 last 99.51) - open/close 36468 (time 1753746466.93 total 396.55 last 95.85) - open/close 37420 (time 1753746476.93 total 406.55 last 95.16) - open/close 38377 (time 1753746486.94 total 416.56 last 95.59) - open/close 39311 (time 1753746496.95 total 426.57 last 93.36) - open/close 40000 (time 1753746503.99 total 433.61 last 97.83) - open/close 40880 (time 1753746514.00 total 443.62 last 87.90) - open/close 41771 (time 1753746524.01 total 453.63 last 89.00) - open/close 42631 (time 1753746534.03 total 463.65 last 85.85) - open/close 43516 (time 1753746544.04 total 473.66 last 88.40) - open/close 44438 (time 1753746554.05 total 483.67 last 92.13) - open/close 45403 (time 1753746564.06 total 493.68 last 96.43) - open/close 46488 (time 1753746574.06 total 503.68 last 108.47) - open/close 47502 (time 1753746584.06 total 513.68 last 101.34) - open/close 48521 (time 1753746594.07 total 523.69 last 101.80) - open/close 49539 (time 1753746604.08 total 533.70 last 101.73) - open/close 50000 (time 1753746608.87 total 538.49 last 96.27) - open/close 51009 (time 1753746618.87 total 548.49 last 100.84) - open/close 52019 (time 1753746628.88 total 558.50 last 100.95) - open/close 53105 (time 1753746638.89 total 568.51 last 108.47) - open/close 54049 (time 1753746648.90 total 578.52 last 94.36) - open/close 54953 (time 1753746658.90 total 588.52 last 90.36) - open/close 56004 (time 1753746668.90 total 598.52 last 105.07) - open/close 56942 (time 1753746678.91 total 608.53 last 93.71) - open/close 57902 (time 1753746688.92 total 618.54 last 95.91) - open/close 59002 (time 1753746698.92 total 628.54 last 109.99) - open/close 60000 (time 1753746706.63 total 636.25 last 129.55) - open/close 61410 (time 1753746716.63 total 646.25 last 140.90) - open/close 62669 (time 1753746726.63 total 656.26 last 125.89) - open/close 63817 (time 1753746736.64 total 666.26 last 114.72) total: 64768 open/close in 674.46 seconds: 96.03 ops/second - /unlink 1182 (time 1753746757.08 total 10.01 last 118.10) - /unlink 2472 (time 1753746767.08 total 20.01 last 128.93) - /unlink 3694 (time 1753746777.08 total 30.02 last 122.15) - /unlink 4770 (time 1753746787.09 total 40.03 last 107.49) - /unlink 5970 (time 1753746797.10 total 50.04 last 119.90) - /unlink 7075 (time 1753746807.11 total 60.05 last 110.39) - /unlink 8166 (time 1753746817.11 total 70.05 last 109.10) - /unlink 9316 (time 1753746827.12 total 80.06 last 114.88) - /unlink 10000 (time 1753746832.86 total 85.79 last 119.31) - /unlink 11062 (time 1753746842.86 total 95.79 last 106.16) - /unlink 12041 (time 1753746852.88 total 105.81 last 97.73) - /unlink 13145 (time 1753746862.89 total 115.82 last 110.31) - /unlink 14370 (time 1753746872.90 total 125.83 last 122.30) - /unlink 15793 (time 1753746882.91 total 135.85 last 142.14) - /unlink 16962 (time 1753746892.93 total 145.86 last 116.76) - /unlink 18175 (time 1753746902.94 total 155.87 last 121.11) - /unlink 19324 (time 1753746912.95 total 165.88 last 114.85) - /unlink 20000 (time 1753746918.35 total 171.28 last 125.06) - /unlink 21322 (time 1753746928.36 total 181.29 last 132.10) - /unlink 22685 (time 1753746938.36 total 191.30 last 136.22) - /unlink 23978 (time 1753746948.37 total 201.30 last 129.21) - /unlink 25333 (time 1753746958.38 total 211.31 last 135.40) - /unlink 26587 (time 1753746968.38 total 221.31 last 125.36) - /unlink 27694 (time 1753746978.38 total 231.31 last 110.69) - /unlink 28840 (time 1753746988.39 total 241.32 last 114.55) - /unlink 30000 (time 1753746997.86 total 250.79 last 122.49) - /unlink 31889 (time 1753747007.86 total 260.79 last 188.90) - /unlink 34247 (time 1753747017.86 total 270.79 last 235.77) - /unlink 36879 (time 1753747027.86 total 280.79 last 263.19) - /unlink 39497 (time 1753747037.86 total 290.79 last 261.77) - /unlink 40000 (time 1753747040.07 total 293.00 last 227.62) - /unlink 42183 (time 1753747050.07 total 303.00 last 218.24) - /unlink 44616 (time 1753747060.07 total 313.01 last 243.27) - /unlink 47184 (time 1753747070.08 total 323.01 last 256.72) - /unlink 49559 (time 1753747080.08 total 333.01 last 237.38) - /unlink 50000 (time 1753747082.10 total 335.03 last 218.54) - /unlink 52432 (time 1753747092.10 total 345.03 last 243.16) - /unlink 54905 (time 1753747102.10 total 355.03 last 247.27) - /unlink 57411 (time 1753747112.11 total 365.04 last 250.51) - /unlink 60000 (time 1753747120.83 total 373.76 last 296.78) - /unlink 62837 (time 1753747130.83 total 383.76 last 283.64) total: 64768 /unlink in 391.99 seconds: 165.23 ops/second Waiting for MDT destroys to complete oleg336-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 - open/close 1643 (time 1753747163.18 total 10.00 last 164.25) - open/close 3183 (time 1753747173.19 total 20.01 last 153.93) - open/close 4497 (time 1753747183.19 total 30.01 last 131.35) - open/close 5636 (time 1753747193.20 total 40.02 last 113.79) - open/close 6807 (time 1753747203.21 total 50.03 last 117.02) - open/close 8040 (time 1753747213.21 total 60.03 last 123.29) - open/close 9360 (time 1753747223.21 total 70.03 last 131.98) - open/close 10000 (time 1753747228.31 total 75.13 last 125.49) - open/close 11323 (time 1753747238.31 total 85.13 last 132.29) - open/close 12676 (time 1753747248.31 total 95.14 last 135.22) - open/close 14090 (time 1753747258.32 total 105.14 last 141.32) - open/close 15482 (time 1753747268.32 total 115.14 last 139.18) - open/close 16749 (time 1753747278.32 total 125.15 last 126.66) - open/close 17997 (time 1753747288.33 total 135.15 last 124.71) - open/close 19350 (time 1753747298.33 total 145.15 last 135.29) - open/close 20000 (time 1753747302.60 total 149.43 last 152.18) - open/close 21516 (time 1753747312.61 total 159.43 last 151.50) - open/close 22946 (time 1753747322.61 total 169.43 last 143.00) - open/close 24555 (time 1753747332.61 total 179.44 last 160.84) - open/close 26239 (time 1753747342.61 total 189.44 last 168.39) - open/close 27968 (time 1753747352.62 total 199.44 last 172.88) - open/close 30000 (time 1753747362.35 total 209.18 last 208.66) - open/close 32158 (time 1753747372.36 total 219.18 last 215.75) - open/close 34044 (time 1753747382.36 total 229.18 last 188.56) - open/close 35870 (time 1753747392.37 total 239.19 last 182.41) - open/close 37873 (time 1753747402.37 total 249.20 last 200.21) - open/close 39915 (time 1753747412.37 total 259.20 last 204.17) - open/close 42029 (time 1753747422.38 total 269.20 last 211.33) - open/close 44018 (time 1753747432.38 total 279.20 last 198.87) - open/close 46328 (time 1753747442.38 total 289.20 last 230.94) - open/close 48792 (time 1753747452.38 total 299.21 last 246.38) - open/close 50000 (time 1753747457.11 total 303.93 last 255.83) - open/close 51974 (time 1753747467.11 total 313.93 last 197.33) - open/close 54122 (time 1753747477.11 total 323.93 last 214.73) - open/close 56369 (time 1753747487.11 total 333.94 last 224.64) - open/close 58757 (time 1753747497.12 total 343.94 last 238.74) - open/close 60000 (time 1753747500.95 total 347.78 last 323.92) - open/close 63021 (time 1753747510.96 total 357.78 last 302.04) total: 64768 open/close in 363.44 seconds: 178.21 ops/second - /unlink 4270 (time 1753747528.22 total 10.00 last 426.94) - /unlink 8806 (time 1753747538.22 total 20.00 last 453.56) - /unlink 10000 (time 1753747540.81 total 22.59 last 461.07) - /unlink 14500 (time 1753747550.81 total 32.59 last 449.89) - /unlink 19707 (time 1753747560.81 total 42.60 last 520.65) - /unlink 25634 (time 1753747570.82 total 52.60 last 592.69) - /unlink 30000 (time 1753747577.12 total 58.90 last 692.37) - /unlink 35521 (time 1753747587.12 total 68.90 last 552.08) - /unlink 40000 (time 1753747595.43 total 77.21 last 538.81) - /unlink 45829 (time 1753747605.43 total 87.21 last 582.90) - /unlink 50000 (time 1753747613.73 total 95.51 last 502.85) - /unlink 54662 (time 1753747623.74 total 105.52 last 465.80) - /unlink 59352 (time 1753747633.74 total 115.52 last 468.98) - /unlink 64030 (time 1753747643.74 total 125.52 last 467.78) total: 64768 /unlink in 126.82 seconds: 510.73 ops/second Waiting for MDT destroys to complete oleg336-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 - open/close 2437 (time 1753747666.13 total 10.00 last 243.64) - open/close 4808 (time 1753747676.13 total 20.00 last 237.04) - open/close 6902 (time 1753747686.13 total 30.01 last 209.35) - open/close 9155 (time 1753747696.14 total 40.01 last 225.22) - open/close 10000 (time 1753747699.08 total 42.96 last 286.83) - open/close 13239 (time 1753747709.08 total 52.96 last 323.85) - open/close 16096 (time 1753747719.09 total 62.96 last 285.66) - open/close 19102 (time 1753747729.09 total 72.96 last 300.55) - open/close 20000 (time 1753747731.86 total 75.73 last 324.37) - open/close 22610 (time 1753747741.86 total 85.73 last 260.92) - open/close 25473 (time 1753747751.86 total 95.73 last 286.27) - open/close 27856 (time 1753747761.86 total 105.74 last 238.26) - open/close 30000 (time 1753747769.47 total 113.35 last 281.76) - open/close 31966 (time 1753747779.47 total 123.35 last 196.56) - open/close 34019 (time 1753747789.47 total 133.35 last 205.29) - open/close 36048 (time 1753747799.48 total 143.35 last 202.81) - open/close 38107 (time 1753747809.48 total 153.35 last 205.88) - open/close 40000 (time 1753747817.22 total 161.09 last 244.51) - open/close 42495 (time 1753747827.22 total 171.10 last 249.44) - open/close 45041 (time 1753747837.22 total 181.10 last 254.59) - open/close 47748 (time 1753747847.22 total 191.10 last 270.68) - open/close 49877 (time 1753747857.23 total 201.10 last 212.85) - open/close 52691 (time 1753747867.23 total 211.10 last 281.33) - open/close 55765 (time 1753747877.23 total 221.10 last 307.35) - open/close 59002 (time 1753747887.23 total 231.11 last 323.60) - open/close 60000 (time 1753747890.97 total 234.84 last 267.10) - open/close 62880 (time 1753747900.97 total 244.85 last 287.94) total: 64768 open/close in 250.42 seconds: 258.63 ops/second - /unlink 5921 (time 1753747918.18 total 10.00 last 592.04) - /unlink 10000 (time 1753747926.23 total 18.05 last 506.57) - /unlink 15345 (time 1753747936.24 total 28.06 last 534.40) - /unlink 20000 (time 1753747945.69 total 37.51 last 492.32) - /unlink 25392 (time 1753747955.70 total 47.52 last 538.61) - /unlink 30000 (time 1753747964.55 total 56.36 last 521.05) - /unlink 36207 (time 1753747974.55 total 66.37 last 620.64) - /unlink 40000 (time 1753747980.31 total 72.12 last 658.69) - /unlink 45880 (time 1753747990.31 total 82.13 last 587.95) - /unlink 50000 (time 1753747997.24 total 89.06 last 594.51) - /unlink 56241 (time 1753748007.24 total 99.06 last 624.09) - /unlink 60000 (time 1753748012.03 total 103.85 last 784.54) total: 64768 /unlink in 110.53 seconds: 585.98 ops/second Waiting for MDT destroys to complete oleg336-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local oleg336-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg336-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2