== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:47:50 (1764733670) Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) oleg152-server: oleg152-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg152-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg152-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 900 (time 1764733788.31 total 10.01 last 89.94) - open/close 1815 (time 1764733798.31 total 20.01 last 91.43) - open/close 2667 (time 1764733808.32 total 30.02 last 85.17) - open/close 3514 (time 1764733818.33 total 40.03 last 84.59) - open/close 4388 (time 1764733828.34 total 50.04 last 87.34) - open/close 5272 (time 1764733838.34 total 60.04 last 88.35) - open/close 6095 (time 1764733848.34 total 70.05 last 82.27) - open/close 6922 (time 1764733858.35 total 80.06 last 82.62) - open/close 7804 (time 1764733868.36 total 90.06 last 88.18) - open/close 8697 (time 1764733878.36 total 100.06 last 89.30) - open/close 9580 (time 1764733888.36 total 110.06 last 88.24) - open/close 10000 (time 1764733892.94 total 114.64 last 91.82) - open/close 10815 (time 1764733902.95 total 124.65 last 81.40) - open/close 11703 (time 1764733912.95 total 134.65 last 88.80) - open/close 12479 (time 1764733922.95 total 144.65 last 77.58) - open/close 13206 (time 1764733932.96 total 154.66 last 72.63) - open/close 14135 (time 1764733942.97 total 164.67 last 92.84) - open/close 15061 (time 1764733952.97 total 174.68 last 92.54) - open/close 15931 (time 1764733962.98 total 184.68 last 86.94) - open/close 16823 (time 1764733972.99 total 194.69 last 89.17) - open/close 17789 (time 1764733982.99 total 204.69 last 96.55) - open/close 18772 (time 1764733992.99 total 214.69 last 98.29) - open/close 19696 (time 1764734003.00 total 224.70 last 92.34) - open/close 20000 (time 1764734006.41 total 228.11 last 89.14) - open/close 20919 (time 1764734016.41 total 238.11 last 91.88) - open/close 21837 (time 1764734026.41 total 248.12 last 91.77) - open/close 22779 (time 1764734036.42 total 258.12 last 94.18) - open/close 23697 (time 1764734046.43 total 268.13 last 91.70) - open/close 24661 (time 1764734056.43 total 278.13 last 96.34) - open/close 25626 (time 1764734066.44 total 288.14 last 96.42) - open/close 26562 (time 1764734076.44 total 298.15 last 93.56) - open/close 27490 (time 1764734086.44 total 308.15 last 92.80) - open/close 28443 (time 1764734096.45 total 318.15 last 95.28) - open/close 29338 (time 1764734106.45 total 328.15 last 89.46) - open/close 30000 (time 1764734113.71 total 335.41 last 91.26) - open/close 30980 (time 1764734123.72 total 345.42 last 97.89) - open/close 31877 (time 1764734133.72 total 355.43 last 89.64) - open/close 32738 (time 1764734143.74 total 365.44 last 85.99) - open/close 33533 (time 1764734153.74 total 375.44 last 79.46) - open/close 34105 (time 1764734163.76 total 385.46 last 57.10) - open/close 34640 (time 1764734173.77 total 395.47 last 53.45) - open/close 35361 (time 1764734183.78 total 405.48 last 72.00) - open/close 36158 (time 1764734193.79 total 415.49 last 79.63) - open/close 36987 (time 1764734203.80 total 425.50 last 82.85) - open/close 37778 (time 1764734213.80 total 435.50 last 79.06) - open/close 38508 (time 1764734223.80 total 445.51 last 72.99) - open/close 39361 (time 1764734233.81 total 455.51 last 85.26) - open/close 40000 (time 1764734241.05 total 462.75 last 88.24) - open/close 40852 (time 1764734251.06 total 472.76 last 85.14) - open/close 41699 (time 1764734261.06 total 482.76 last 84.66) - open/close 42537 (time 1764734271.07 total 492.77 last 83.77) - open/close 43389 (time 1764734281.09 total 502.79 last 85.03) - open/close 44275 (time 1764734291.09 total 512.80 last 88.53) - open/close 45152 (time 1764734301.10 total 522.80 last 87.66) - open/close 46014 (time 1764734311.10 total 532.80 last 86.17) - open/close 46884 (time 1764734321.11 total 542.81 last 86.91) - open/close 47724 (time 1764734331.13 total 552.83 last 83.88) - open/close 48441 (time 1764734341.13 total 562.83 last 71.68) - open/close 49310 (time 1764734351.13 total 572.84 last 86.86) - open/close 50000 (time 1764734359.48 total 581.18 last 82.72) - open/close 50818 (time 1764734369.48 total 591.18 last 81.75) - open/close 51668 (time 1764734379.49 total 601.19 last 84.94) - open/close 52486 (time 1764734389.50 total 611.20 last 81.71) - open/close 53358 (time 1764734399.51 total 621.21 last 87.16) - open/close 54180 (time 1764734409.51 total 631.21 last 82.17) - open/close 55001 (time 1764734419.51 total 641.21 last 82.08) - open/close 55833 (time 1764734429.51 total 651.21 last 83.20) - open/close 56691 (time 1764734439.52 total 661.22 last 85.70) - open/close 57512 (time 1764734449.52 total 671.23 last 82.09) - open/close 58295 (time 1764734459.55 total 681.25 last 78.11) - open/close 59110 (time 1764734469.56 total 691.26 last 81.42) - open/close 59964 (time 1764734479.56 total 701.27 last 85.35) - open/close 60823 (time 1764734489.57 total 711.27 last 85.85) - open/close 61644 (time 1764734499.58 total 721.28 last 82.05) - open/close 62392 (time 1764734509.59 total 731.29 last 74.69) - open/close 63276 (time 1764734519.59 total 741.29 last 88.40) - open/close 64123 (time 1764734529.60 total 751.30 last 84.65) total: 64768 open/close in 759.01 seconds: 85.33 ops/second - /unlink 916 (time 1764734550.48 total 10.01 last 91.54) - /unlink 1681 (time 1764734560.49 total 20.01 last 76.44) - /unlink 2505 (time 1764734570.49 total 30.02 last 82.34) - /unlink 3307 (time 1764734580.50 total 40.03 last 80.12) - /unlink 4151 (time 1764734590.51 total 50.04 last 84.35) - /unlink 5005 (time 1764734600.51 total 60.04 last 85.37) - /unlink 5879 (time 1764734610.52 total 70.04 last 87.37) - /unlink 6716 (time 1764734620.52 total 80.05 last 83.65) - /unlink 7529 (time 1764734630.53 total 90.05 last 81.27) - /unlink 8406 (time 1764734640.54 total 100.07 last 87.58) - /unlink 9306 (time 1764734650.55 total 110.07 last 89.96) - /unlink 10000 (time 1764734658.44 total 117.97 last 87.91) - /unlink 10753 (time 1764734668.45 total 127.98 last 75.23) - /unlink 11598 (time 1764734678.45 total 137.98 last 84.47) - /unlink 12438 (time 1764734688.46 total 147.99 last 83.94) - /unlink 13236 (time 1764734698.46 total 157.99 last 79.79) - /unlink 14051 (time 1764734708.46 total 167.99 last 81.46) - /unlink 14858 (time 1764734718.47 total 178.00 last 80.63) - /unlink 15762 (time 1764734728.48 total 188.01 last 90.30) - /unlink 16634 (time 1764734738.50 total 198.03 last 87.08) - /unlink 17465 (time 1764734748.53 total 208.06 last 82.84) - /unlink 18395 (time 1764734758.54 total 218.06 last 92.96) - /unlink 19242 (time 1764734768.54 total 228.07 last 84.65) - /unlink 20000 (time 1764734777.54 total 237.07 last 84.21) - /unlink 20840 (time 1764734787.55 total 247.08 last 83.94) - /unlink 21706 (time 1764734797.56 total 257.09 last 86.49) - /unlink 22548 (time 1764734807.56 total 267.09 last 84.20) - /unlink 23487 (time 1764734817.57 total 277.09 last 93.87) - /unlink 24492 (time 1764734827.58 total 287.11 last 100.34) - /unlink 25572 (time 1764734837.59 total 297.12 last 107.86) - /unlink 26651 (time 1764734847.60 total 307.13 last 107.83) - /unlink 27676 (time 1764734857.61 total 317.13 last 102.44) - /unlink 28705 (time 1764734867.61 total 327.14 last 102.85) - /unlink 29795 (time 1764734877.62 total 337.14 last 108.95) - /unlink 30000 (time 1764734879.66 total 339.19 last 100.15) - /unlink 31119 (time 1764734889.67 total 349.19 last 111.87) - /unlink 32106 (time 1764734899.67 total 359.20 last 98.62) - /unlink 33048 (time 1764734909.68 total 369.20 last 94.18) - /unlink 33971 (time 1764734919.68 total 379.21 last 92.28) - /unlink 34908 (time 1764734929.68 total 389.21 last 93.67) - /unlink 35980 (time 1764734939.69 total 399.21 last 107.15) - /unlink 36953 (time 1764734949.69 total 409.22 last 97.27) - /unlink 37981 (time 1764734959.69 total 419.22 last 102.76) - /unlink 38933 (time 1764734969.70 total 429.23 last 95.10) - /unlink 39850 (time 1764734979.70 total 439.23 last 91.70) - /unlink 40770 (time 1764734989.71 total 449.24 last 91.94) - /unlink 41745 (time 1764734999.71 total 459.24 last 97.45) - /unlink 42667 (time 1764735009.72 total 469.25 last 92.17) - /unlink 43729 (time 1764735019.72 total 479.25 last 106.14) - /unlink 44621 (time 1764735029.73 total 489.26 last 89.16) - /unlink 45510 (time 1764735039.73 total 499.26 last 88.88) - /unlink 46543 (time 1764735049.73 total 509.26 last 103.28) - /unlink 47693 (time 1764735059.74 total 519.27 last 114.90) - /unlink 48655 (time 1764735069.75 total 529.28 last 96.09) - /unlink 49682 (time 1764735079.76 total 539.28 last 102.66) - /unlink 50000 (time 1764735082.64 total 542.17 last 110.30) - /unlink 51161 (time 1764735092.64 total 552.17 last 116.05) - /unlink 52389 (time 1764735102.64 total 562.17 last 122.80) - /unlink 53720 (time 1764735112.65 total 572.18 last 132.99) - /unlink 54985 (time 1764735122.66 total 582.18 last 126.44) - /unlink 56110 (time 1764735132.67 total 592.19 last 112.39) - /unlink 57409 (time 1764735142.67 total 602.20 last 129.79) - /unlink 58636 (time 1764735152.68 total 612.20 last 122.69) - /unlink 59855 (time 1764735162.68 total 622.21 last 121.86) - /unlink 61246 (time 1764735172.68 total 632.21 last 139.02) - /unlink 62472 (time 1764735182.69 total 642.22 last 122.49) - /unlink 63556 (time 1764735192.70 total 652.22 last 108.37) total: 64768 /unlink in 661.40 seconds: 97.93 ops/second Waiting for MDT destroys to complete oleg152-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 - open/close 1071 (time 1764735222.63 total 10.00 last 107.10) - open/close 2641 (time 1764735232.63 total 20.00 last 156.99) - open/close 4039 (time 1764735242.63 total 30.00 last 139.75) - open/close 5416 (time 1764735252.64 total 40.01 last 137.60) - open/close 6841 (time 1764735262.65 total 50.02 last 142.39) - open/close 8132 (time 1764735272.65 total 60.02 last 129.02) - open/close 9432 (time 1764735282.66 total 70.03 last 129.95) - open/close 10000 (time 1764735286.24 total 73.61 last 158.39) - open/close 11697 (time 1764735296.25 total 83.62 last 169.63) - open/close 13250 (time 1764735306.25 total 93.62 last 155.28) - open/close 15106 (time 1764735316.25 total 103.62 last 185.53) - open/close 16563 (time 1764735326.26 total 113.63 last 145.62) - open/close 17620 (time 1764735336.26 total 123.63 last 105.65) - open/close 18490 (time 1764735346.27 total 133.64 last 86.94) - open/close 19245 (time 1764735356.28 total 143.65 last 75.44) - open/close 20000 (time 1764735366.25 total 153.62 last 75.69) - open/close 20895 (time 1764735376.26 total 163.63 last 89.41) - open/close 21922 (time 1764735386.27 total 173.64 last 102.64) - open/close 22777 (time 1764735396.28 total 183.65 last 85.43) - open/close 23640 (time 1764735406.28 total 193.65 last 86.26) - open/close 24453 (time 1764735416.29 total 203.66 last 81.23) - open/close 25276 (time 1764735426.29 total 213.66 last 82.29) - open/close 26112 (time 1764735436.30 total 223.67 last 83.54) - open/close 26943 (time 1764735446.31 total 233.68 last 82.99) - open/close 27874 (time 1764735456.32 total 243.69 last 93.02) - open/close 28672 (time 1764735466.33 total 253.70 last 79.77) - open/close 29517 (time 1764735476.33 total 263.70 last 84.43) - open/close 30000 (time 1764735482.63 total 270.00 last 76.73) - open/close 30811 (time 1764735492.63 total 280.01 last 81.05) - open/close 31638 (time 1764735502.64 total 290.01 last 82.68) - open/close 32464 (time 1764735512.64 total 300.01 last 82.54) - open/close 33323 (time 1764735522.65 total 310.02 last 85.83) - open/close 34261 (time 1764735532.66 total 320.03 last 93.73) - open/close 35168 (time 1764735542.67 total 330.04 last 90.62) - open/close 36061 (time 1764735552.67 total 340.04 last 89.27) - open/close 36977 (time 1764735562.67 total 350.05 last 91.57) - open/close 37829 (time 1764735572.68 total 360.05 last 85.12) - open/close 38723 (time 1764735582.69 total 370.06 last 89.36) - open/close 39565 (time 1764735592.69 total 380.06 last 84.18) - open/close 40000 (time 1764735596.79 total 384.16 last 106.00) - open/close 41081 (time 1764735606.80 total 394.17 last 108.01) - open/close 42232 (time 1764735616.81 total 404.18 last 115.02) - open/close 43485 (time 1764735626.81 total 414.18 last 125.29) - open/close 44733 (time 1764735636.82 total 424.19 last 124.74) - open/close 45927 (time 1764735646.82 total 434.19 last 119.36) - open/close 47445 (time 1764735656.82 total 444.19 last 151.72) - open/close 48843 (time 1764735666.83 total 454.20 last 139.76) - open/close 50000 (time 1764735675.10 total 462.47 last 139.91) - open/close 51468 (time 1764735685.10 total 472.47 last 146.74) - open/close 52986 (time 1764735695.10 total 482.47 last 151.76) - open/close 54384 (time 1764735705.11 total 492.48 last 139.73) - open/close 55903 (time 1764735715.12 total 502.49 last 151.76) - open/close 57363 (time 1764735725.12 total 512.50 last 145.88) - open/close 58924 (time 1764735735.13 total 522.50 last 156.10) - open/close 60000 (time 1764735741.52 total 528.89 last 168.17) - open/close 61455 (time 1764735751.53 total 538.90 last 145.44) - open/close 62880 (time 1764735761.53 total 548.90 last 142.46) - open/close 64104 (time 1764735771.54 total 558.91 last 122.24) total: 64768 open/close in 564.57 seconds: 114.72 ops/second - /unlink 2039 (time 1764735789.41 total 10.00 last 203.84) - /unlink 4263 (time 1764735799.41 total 20.01 last 222.35) - /unlink 6664 (time 1764735809.41 total 30.01 last 240.06) - /unlink 9172 (time 1764735819.42 total 40.01 last 250.75) - /unlink 10000 (time 1764735823.41 total 44.01 last 207.11) - /unlink 12003 (time 1764735833.42 total 54.01 last 200.22) - /unlink 14257 (time 1764735843.42 total 64.01 last 225.34) - /unlink 16291 (time 1764735853.42 total 74.01 last 203.39) - /unlink 18261 (time 1764735863.42 total 84.01 last 197.00) - /unlink 20000 (time 1764735870.96 total 91.55 last 230.81) - /unlink 22301 (time 1764735880.96 total 101.55 last 230.00) - /unlink 24088 (time 1764735890.96 total 111.55 last 178.69) - /unlink 26348 (time 1764735900.96 total 121.56 last 225.95) - /unlink 28481 (time 1764735910.97 total 131.57 last 213.08) - /unlink 30000 (time 1764735917.86 total 138.45 last 220.51) - /unlink 32370 (time 1764735927.86 total 148.46 last 236.94) - /unlink 34771 (time 1764735937.87 total 158.46 last 239.98) - /unlink 37238 (time 1764735947.87 total 168.46 last 246.63) - /unlink 39733 (time 1764735957.88 total 178.47 last 249.40) - /unlink 42065 (time 1764735967.88 total 188.47 last 233.07) - /unlink 44170 (time 1764735977.89 total 198.48 last 210.44) - /unlink 46568 (time 1764735987.89 total 208.48 last 239.78) - /unlink 48654 (time 1764735997.89 total 218.48 last 208.55) - /unlink 50000 (time 1764736004.36 total 224.96 last 207.85) - /unlink 52538 (time 1764736014.37 total 234.96 last 253.69) - /unlink 54953 (time 1764736024.37 total 244.96 last 241.45) - /unlink 57368 (time 1764736034.37 total 254.97 last 241.42) - /unlink 59680 (time 1764736044.38 total 264.97 last 231.11) - /unlink 62024 (time 1764736054.38 total 274.97 last 234.39) - /unlink 64162 (time 1764736064.38 total 284.97 last 213.78) total: 64768 /unlink in 287.42 seconds: 225.34 ops/second Waiting for MDT destroys to complete oleg152-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 - open/close 1174 (time 1764736087.52 total 10.00 last 117.38) - open/close 2451 (time 1764736097.53 total 20.01 last 127.61) - open/close 3672 (time 1764736107.53 total 30.01 last 122.09) - open/close 4855 (time 1764736117.53 total 40.01 last 118.30) - open/close 5980 (time 1764736127.54 total 50.02 last 112.42) - open/close 7202 (time 1764736137.54 total 60.02 last 122.18) - open/close 8564 (time 1764736147.54 total 70.02 last 136.19) - open/close 9632 (time 1764736157.55 total 80.03 last 106.67) - open/close 10000 (time 1764736162.88 total 85.35 last 69.15) - open/close 10843 (time 1764736172.89 total 95.37 last 84.14) - open/close 11681 (time 1764736182.90 total 105.38 last 83.77) - open/close 12569 (time 1764736192.90 total 115.38 last 88.74) - open/close 13419 (time 1764736202.91 total 125.39 last 84.97) - open/close 14337 (time 1764736212.92 total 135.39 last 91.72) - open/close 15203 (time 1764736222.92 total 145.40 last 86.54) - open/close 16080 (time 1764736232.93 total 155.41 last 87.61) - open/close 16931 (time 1764736242.94 total 165.42 last 85.04) - open/close 17874 (time 1764736252.95 total 175.43 last 94.22) - open/close 18799 (time 1764736262.95 total 185.43 last 92.49) - open/close 19703 (time 1764736272.96 total 195.43 last 90.34) - open/close 20000 (time 1764736276.10 total 198.58 last 94.45) - open/close 20923 (time 1764736286.11 total 208.59 last 92.19) - open/close 21821 (time 1764736296.12 total 218.60 last 89.70) - open/close 22704 (time 1764736306.13 total 228.60 last 88.27) - open/close 23608 (time 1764736316.13 total 238.61 last 90.39) - open/close 24567 (time 1764736326.13 total 248.61 last 95.85) - open/close 25483 (time 1764736336.14 total 258.61 last 91.57) - open/close 26513 (time 1764736346.15 total 268.62 last 102.90) - open/close 27469 (time 1764736356.15 total 278.63 last 95.54) - open/close 28366 (time 1764736366.16 total 288.64 last 89.61) - open/close 29268 (time 1764736376.17 total 298.65 last 90.16) - open/close 30000 (time 1764736384.57 total 307.05 last 87.13) - open/close 30856 (time 1764736394.57 total 317.05 last 85.60) - open/close 31758 (time 1764736404.58 total 327.05 last 90.13) - open/close 32723 (time 1764736414.58 total 337.06 last 96.47) - open/close 33736 (time 1764736424.58 total 347.06 last 101.29) - open/close 34587 (time 1764736434.58 total 357.06 last 85.08) - open/close 35484 (time 1764736444.59 total 367.06 last 89.66) - open/close 36365 (time 1764736454.59 total 377.07 last 88.08) - open/close 37260 (time 1764736464.59 total 387.07 last 89.47) - open/close 38129 (time 1764736474.60 total 397.08 last 86.85) - open/close 39033 (time 1764736484.60 total 407.08 last 90.35) - open/close 39899 (time 1764736494.60 total 417.08 last 86.59) - open/close 40834 (time 1764736504.61 total 427.09 last 93.46) - open/close 41775 (time 1764736514.62 total 437.10 last 93.97) - open/close 42727 (time 1764736524.62 total 447.10 last 95.20) - open/close 43677 (time 1764736534.63 total 457.11 last 94.92) - open/close 44622 (time 1764736544.64 total 467.12 last 94.39) - open/close 45527 (time 1764736554.65 total 477.13 last 90.44) - open/close 46486 (time 1764736564.65 total 487.13 last 95.89) - open/close 47445 (time 1764736574.67 total 497.15 last 95.74) - open/close 48301 (time 1764736584.68 total 507.16 last 85.50) - open/close 49253 (time 1764736594.68 total 517.16 last 95.19) - open/close 50000 (time 1764736602.06 total 524.53 last 101.29) - open/close 50945 (time 1764736612.06 total 534.53 last 94.50) - open/close 51874 (time 1764736622.06 total 544.54 last 92.86) - open/close 52807 (time 1764736632.07 total 554.55 last 93.19) - open/close 53822 (time 1764736642.07 total 564.55 last 101.47) - open/close 54893 (time 1764736652.08 total 574.56 last 107.06) - open/close 55895 (time 1764736662.09 total 584.56 last 100.12) - open/close 56854 (time 1764736672.10 total 594.58 last 95.80) - open/close 57755 (time 1764736682.10 total 604.58 last 90.08) - open/close 58659 (time 1764736692.11 total 614.59 last 90.33) - open/close 59574 (time 1764736702.12 total 624.60 last 91.40) - open/close 60000 (time 1764736707.00 total 629.48 last 87.23) - open/close 60927 (time 1764736717.02 total 639.50 last 92.55) - open/close 61588 (time 1764736727.02 total 649.50 last 66.08) - open/close 62154 (time 1764736737.02 total 659.50 last 56.57) - open/close 62711 (time 1764736747.04 total 669.52 last 55.62) - open/close 63298 (time 1764736757.05 total 679.53 last 58.63) - open/close 64154 (time 1764736767.05 total 689.53 last 85.58) total: 64768 open/close in 697.02 seconds: 92.92 ops/second - /unlink 797 (time 1764736788.51 total 10.01 last 79.61) - /unlink 1600 (time 1764736798.52 total 20.02 last 80.23) - /unlink 2445 (time 1764736808.53 total 30.03 last 84.43) - /unlink 3295 (time 1764736818.54 total 40.04 last 84.94) - /unlink 4165 (time 1764736828.55 total 50.04 last 86.93) - /unlink 5026 (time 1764736838.55 total 60.05 last 86.08) - /unlink 5875 (time 1764736848.56 total 70.06 last 84.81) - /unlink 6724 (time 1764736858.57 total 80.06 last 84.83) - /unlink 7612 (time 1764736868.58 total 90.08 last 88.67) - /unlink 8483 (time 1764736878.59 total 100.09 last 87.02) - /unlink 9323 (time 1764736888.60 total 110.09 last 83.96) - /unlink 10000 (time 1764736896.69 total 118.18 last 83.67) - /unlink 10845 (time 1764736906.70 total 128.19 last 84.43) - /unlink 11636 (time 1764736916.70 total 138.20 last 79.07) - /unlink 12456 (time 1764736926.70 total 148.20 last 81.98) - /unlink 13275 (time 1764736936.71 total 158.21 last 81.85) - /unlink 14094 (time 1764736946.71 total 168.21 last 81.87) - /unlink 14939 (time 1764736956.72 total 178.22 last 84.45) - /unlink 15850 (time 1764736966.72 total 188.22 last 91.07) - /unlink 16790 (time 1764736976.73 total 198.23 last 93.92) - /unlink 17707 (time 1764736986.73 total 208.23 last 91.66) - /unlink 18667 (time 1764736996.73 total 218.23 last 96.00) - /unlink 19546 (time 1764737006.73 total 228.23 last 87.90) - /unlink 20000 (time 1764737011.65 total 233.14 last 92.43) - /unlink 20981 (time 1764737021.66 total 243.15 last 97.99) - /unlink 21879 (time 1764737031.66 total 253.16 last 89.74) - /unlink 22706 (time 1764737041.67 total 263.17 last 82.62) - /unlink 23607 (time 1764737051.67 total 273.17 last 90.09) - /unlink 24479 (time 1764737061.68 total 283.18 last 87.17) - /unlink 25497 (time 1764737071.68 total 293.18 last 101.74) - /unlink 26622 (time 1764737081.69 total 303.18 last 112.47) - /unlink 27595 (time 1764737091.69 total 313.19 last 97.27) - /unlink 28378 (time 1764737101.70 total 323.19 last 78.25) - /unlink 29317 (time 1764737111.70 total 333.19 last 93.90) - /unlink 30000 (time 1764737118.66 total 340.16 last 98.04) - /unlink 31110 (time 1764737128.67 total 350.16 last 110.97) - /unlink 32262 (time 1764737138.68 total 360.17 last 115.06) - /unlink 33309 (time 1764737148.68 total 370.18 last 104.64) - /unlink 34411 (time 1764737158.69 total 380.18 last 110.15) - /unlink 35835 (time 1764737168.69 total 390.19 last 142.37) - /unlink 37155 (time 1764737178.69 total 400.19 last 131.96) - /unlink 38298 (time 1764737188.70 total 410.19 last 114.27) - /unlink 39494 (time 1764737198.70 total 420.19 last 119.60) - /unlink 40000 (time 1764737202.53 total 424.03 last 131.79) - /unlink 41316 (time 1764737212.54 total 434.04 last 131.54) - /unlink 42722 (time 1764737222.54 total 444.04 last 140.57) - /unlink 44365 (time 1764737232.54 total 454.04 last 164.28) - /unlink 45617 (time 1764737242.54 total 464.04 last 125.19) - /unlink 47261 (time 1764737252.54 total 474.04 last 164.38) - /unlink 49153 (time 1764737262.55 total 484.04 last 189.17) - /unlink 50000 (time 1764737268.29 total 489.78 last 147.56) - /unlink 51433 (time 1764737278.29 total 499.79 last 143.25) - /unlink 52898 (time 1764737288.29 total 509.79 last 146.47) - /unlink 54859 (time 1764737298.30 total 519.80 last 195.94) - /unlink 57085 (time 1764737308.30 total 529.80 last 222.54) - /unlink 59261 (time 1764737318.31 total 539.80 last 217.54) - /unlink 60000 (time 1764737323.07 total 544.57 last 154.94) - /unlink 61511 (time 1764737333.08 total 554.58 last 151.03) - /unlink 64037 (time 1764737343.08 total 564.58 last 252.58) total: 64768 /unlink in 567.23 seconds: 114.18 ops/second Waiting for MDT destroys to complete oleg152-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 - open/close 2052 (time 1764737367.34 total 10.00 last 205.14) - open/close 3748 (time 1764737377.34 total 20.00 last 169.59) - open/close 5547 (time 1764737387.34 total 30.01 last 179.84) - open/close 7341 (time 1764737397.34 total 40.01 last 179.36) - open/close 9293 (time 1764737407.35 total 50.01 last 195.14) - open/close 10000 (time 1764737410.69 total 53.36 last 211.27) - open/close 11948 (time 1764737420.70 total 63.37 last 194.67) - open/close 13558 (time 1764737430.70 total 73.37 last 160.99) - open/close 15125 (time 1764737440.70 total 83.37 last 156.70) - open/close 16727 (time 1764737450.70 total 93.37 last 160.19) - open/close 18569 (time 1764737460.70 total 103.37 last 184.19) - open/close 20000 (time 1764737468.34 total 111.01 last 187.31) - open/close 21224 (time 1764737478.35 total 121.01 last 122.32) - open/close 22449 (time 1764737488.35 total 131.02 last 122.46) - open/close 23800 (time 1764737498.36 total 141.02 last 134.99) - open/close 24858 (time 1764737508.36 total 151.03 last 105.72) - open/close 25912 (time 1764737518.37 total 161.03 last 105.37) - open/close 27280 (time 1764737528.37 total 171.04 last 136.75) - open/close 28622 (time 1764737538.38 total 181.04 last 134.12) - open/close 29817 (time 1764737548.38 total 191.05 last 119.46) - open/close 31089 (time 1764737558.40 total 201.06 last 127.01) - open/close 32243 (time 1764737568.40 total 211.07 last 115.36) - open/close 33386 (time 1764737578.40 total 221.07 last 114.25) - open/close 34587 (time 1764737588.41 total 231.07 last 120.08) - open/close 36061 (time 1764737598.41 total 241.07 last 147.35) - open/close 36897 (time 1764737608.41 total 251.08 last 83.55) - open/close 38190 (time 1764737618.42 total 261.09 last 129.23) - open/close 39962 (time 1764737628.42 total 271.09 last 177.13) - open/close 41707 (time 1764737638.42 total 281.09 last 174.49) - open/close 43436 (time 1764737648.43 total 291.10 last 172.82) - open/close 45204 (time 1764737658.44 total 301.10 last 176.68) - open/close 46496 (time 1764737668.45 total 311.11 last 129.08) - open/close 47851 (time 1764737678.45 total 321.12 last 135.46) - open/close 49837 (time 1764737688.45 total 331.12 last 198.53) - open/close 51631 (time 1764737698.45 total 341.12 last 179.37) - open/close 53843 (time 1764737708.46 total 351.12 last 221.13) - open/close 55480 (time 1764737718.46 total 361.13 last 163.61) - open/close 56857 (time 1764737728.46 total 371.13 last 137.68) - open/close 58158 (time 1764737738.47 total 381.13 last 130.05) - open/close 59626 (time 1764737748.47 total 391.14 last 146.74) - open/close 60000 (time 1764737750.96 total 393.63 last 150.41) - open/close 61640 (time 1764737760.96 total 403.63 last 163.94) - open/close 63378 (time 1764737770.96 total 413.63 last 173.76) total: 64768 open/close in 423.58 seconds: 152.91 ops/second - /unlink 3050 (time 1764737792.72 total 10.00 last 305.00) - /unlink 6759 (time 1764737802.72 total 20.00 last 370.85) - /unlink 10000 (time 1764737811.32 total 28.60 last 376.95) - /unlink 13436 (time 1764737821.32 total 38.60 last 343.55) - /unlink 17286 (time 1764737831.32 total 48.60 last 384.92) - /unlink 20000 (time 1764737837.97 total 55.25 last 408.09) - /unlink 23869 (time 1764737847.97 total 65.26 last 386.82) - /unlink 27741 (time 1764737857.97 total 75.26 last 387.19) - /unlink 30000 (time 1764737863.60 total 80.88 last 401.51) - /unlink 35134 (time 1764737873.60 total 90.88 last 513.40) - /unlink 39568 (time 1764737883.60 total 100.88 last 443.38) - /unlink 44283 (time 1764737893.60 total 110.89 last 471.24) - /unlink 49658 (time 1764737903.60 total 120.89 last 537.48) - /unlink 54757 (time 1764737913.61 total 130.89 last 509.84) - /unlink 58950 (time 1764737923.61 total 140.89 last 419.22) - /unlink 60000 (time 1764737926.46 total 143.75 last 367.92) - /unlink 63332 (time 1764737936.46 total 153.75 last 333.16) total: 64768 /unlink in 158.03 seconds: 409.85 ops/second Waiting for MDT destroys to complete oleg152-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 - open/close 1827 (time 1764737961.21 total 10.01 last 182.58) - open/close 3828 (time 1764737971.21 total 20.01 last 200.09) - open/close 5921 (time 1764737981.22 total 30.01 last 209.28) - open/close 7662 (time 1764737991.22 total 40.01 last 174.01) - open/close 10000 (time 1764738000.24 total 49.03 last 259.23) - open/close 13239 (time 1764738010.25 total 59.04 last 323.72) - open/close 16040 (time 1764738020.25 total 69.04 last 280.07) - open/close 19469 (time 1764738030.25 total 79.04 last 342.82) - open/close 22820 (time 1764738040.25 total 89.04 last 335.04) - open/close 25655 (time 1764738050.26 total 99.05 last 283.38) - open/close 27702 (time 1764738060.26 total 109.05 last 204.56) - open/close 29764 (time 1764738070.26 total 119.06 last 206.16) - open/close 31630 (time 1764738080.27 total 129.06 last 186.57) - open/close 33143 (time 1764738090.27 total 139.06 last 151.23) - open/close 35515 (time 1764738100.27 total 149.07 last 237.14) - open/close 37255 (time 1764738110.28 total 159.07 last 173.88) - open/close 38887 (time 1764738120.29 total 169.08 last 163.10) - open/close 40000 (time 1764738125.32 total 174.11 last 221.23) - open/close 42702 (time 1764738135.32 total 184.11 last 270.10) - open/close 44551 (time 1764738145.32 total 194.11 last 184.87) - open/close 45895 (time 1764738155.33 total 204.12 last 134.35) - open/close 47406 (time 1764738165.33 total 214.12 last 151.06) - open/close 48565 (time 1764738175.33 total 224.13 last 115.84) - open/close 49835 (time 1764738185.34 total 234.13 last 126.96) - open/close 51222 (time 1764738195.34 total 244.13 last 138.61) - open/close 53189 (time 1764738205.34 total 254.14 last 196.68) - open/close 55436 (time 1764738215.34 total 264.14 last 224.67) - open/close 56944 (time 1764738225.35 total 274.14 last 150.71) - open/close 58514 (time 1764738235.35 total 284.14 last 156.98) - open/close 60000 (time 1764738242.22 total 291.02 last 216.27) - open/close 61889 (time 1764738252.22 total 301.02 last 188.88) - open/close 63596 (time 1764738262.23 total 311.02 last 170.60) total: 64768 open/close in 317.54 seconds: 203.97 ops/second - /unlink 3758 (time 1764738280.42 total 10.00 last 375.75) - /unlink 7029 (time 1764738290.42 total 20.00 last 327.05) - /unlink 10000 (time 1764738296.70 total 26.28 last 473.60) - /unlink 14897 (time 1764738306.70 total 36.28 last 489.67) - /unlink 18999 (time 1764738316.70 total 46.28 last 410.14) - /unlink 20000 (time 1764738318.97 total 48.55 last 440.23) - /unlink 23953 (time 1764738328.97 total 58.55 last 395.23) - /unlink 28358 (time 1764738338.97 total 68.55 last 440.46) - /unlink 30000 (time 1764738342.33 total 71.91 last 489.28) - /unlink 34319 (time 1764738352.33 total 81.91 last 431.76) - /unlink 39217 (time 1764738362.33 total 91.91 last 489.76) - /unlink 45151 (time 1764738372.34 total 101.92 last 593.34) - /unlink 50000 (time 1764738381.66 total 111.24 last 520.22) - /unlink 54879 (time 1764738391.66 total 121.24 last 487.87) - /unlink 59949 (time 1764738401.66 total 131.24 last 506.93) - /unlink 63471 (time 1764738411.66 total 141.24 last 352.06) total: 64768 /unlink in 145.10 seconds: 446.38 ops/second Waiting for MDT destroys to complete oleg152-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2