-----============= acceptance-small: conf-sanity ============----- Thu Nov 13 19:37:48 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg437-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball Stopping clients: oleg437-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg437-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg437-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg437-server oleg437-server: oleg437-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg437-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg437-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg437-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg437-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg437-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:41:22 (1763080882) Stopping clients: oleg437-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg437-client.virtnet /mnt/lustre2 (opts:-f) oleg437-server: oleg437-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg437-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg437-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 804 (time 1763081008.03 total 10.01 last 80.34) - open/close 1580 (time 1763081018.03 total 20.02 last 77.54) - open/close 2326 (time 1763081028.05 total 30.03 last 74.51) - open/close 3120 (time 1763081038.05 total 40.03 last 79.39) - open/close 3884 (time 1763081048.06 total 50.04 last 76.27) - open/close 4572 (time 1763081058.07 total 60.05 last 68.77) - open/close 5424 (time 1763081068.08 total 70.07 last 85.06) - open/close 6069 (time 1763081078.10 total 80.08 last 64.42) - open/close 6719 (time 1763081088.13 total 90.11 last 64.80) - open/close 7322 (time 1763081098.14 total 100.12 last 60.25) - open/close 8029 (time 1763081108.15 total 110.13 last 70.59) - open/close 8678 (time 1763081118.16 total 120.15 last 64.83) - open/close 9328 (time 1763081128.17 total 130.15 last 64.99) - open/close 9864 (time 1763081138.17 total 140.15 last 53.58) - open/close 10000 (time 1763081140.19 total 142.17 last 67.24) - open/close 10624 (time 1763081150.19 total 152.18 last 62.39) - open/close 11440 (time 1763081160.20 total 162.18 last 81.52) - open/close 12242 (time 1763081170.21 total 172.19 last 80.15) - open/close 13162 (time 1763081180.22 total 182.20 last 91.91) - open/close 13983 (time 1763081190.23 total 192.21 last 82.03) - open/close 14824 (time 1763081200.24 total 202.22 last 84.05) - open/close 15740 (time 1763081210.24 total 212.22 last 91.58) - open/close 16574 (time 1763081220.24 total 222.22 last 83.40) - open/close 17421 (time 1763081230.24 total 232.22 last 84.65) - open/close 18321 (time 1763081240.24 total 242.22 last 90.00) - open/close 19191 (time 1763081250.26 total 252.24 last 86.89) - open/close 20000 (time 1763081259.84 total 261.82 last 84.41) - open/close 20848 (time 1763081269.85 total 271.83 last 84.72) - open/close 21744 (time 1763081279.86 total 281.84 last 89.50) - open/close 22576 (time 1763081289.87 total 291.85 last 83.15) - open/close 23456 (time 1763081299.87 total 301.85 last 87.94) - open/close 24309 (time 1763081309.88 total 311.86 last 85.20) - open/close 25127 (time 1763081319.89 total 321.87 last 81.77) - open/close 25972 (time 1763081329.89 total 331.88 last 84.44) - open/close 26852 (time 1763081339.90 total 341.88 last 87.92) - open/close 27762 (time 1763081349.91 total 351.89 last 90.91) - open/close 28595 (time 1763081359.92 total 361.90 last 83.25) - open/close 29345 (time 1763081369.93 total 371.91 last 74.95) - open/close 30000 (time 1763081377.39 total 379.37 last 87.76) - open/close 30892 (time 1763081387.39 total 389.38 last 89.16) - open/close 31732 (time 1763081397.39 total 399.38 last 84.00) - open/close 32572 (time 1763081407.40 total 409.38 last 83.99) - open/close 33455 (time 1763081417.40 total 419.38 last 88.25) - open/close 34324 (time 1763081427.41 total 429.39 last 86.86) - open/close 35220 (time 1763081437.41 total 439.39 last 89.55) - open/close 36010 (time 1763081447.42 total 449.40 last 78.91) - open/close 36739 (time 1763081457.43 total 459.41 last 72.87) - open/close 37558 (time 1763081467.43 total 469.41 last 81.86) - open/close 38437 (time 1763081477.44 total 479.42 last 87.83) - open/close 39375 (time 1763081487.45 total 489.43 last 93.73) - open/close 40000 (time 1763081494.24 total 496.22 last 91.98) - open/close 40863 (time 1763081504.25 total 506.23 last 86.27) - open/close 41728 (time 1763081514.25 total 516.23 last 86.49) - open/close 42628 (time 1763081524.25 total 526.23 last 90.00) - open/close 43436 (time 1763081534.26 total 536.24 last 80.72) - open/close 44239 (time 1763081544.27 total 546.25 last 80.17) - open/close 44974 (time 1763081554.28 total 556.26 last 73.49) - open/close 45880 (time 1763081564.28 total 566.26 last 90.58) - open/close 46687 (time 1763081574.28 total 576.27 last 80.65) - open/close 47565 (time 1763081584.30 total 586.28 last 87.67) - open/close 48407 (time 1763081594.31 total 596.29 last 84.14) - open/close 49297 (time 1763081604.31 total 606.29 last 88.97) - open/close 50000 (time 1763081612.14 total 614.12 last 89.83) - open/close 50867 (time 1763081622.14 total 624.12 last 86.70) - open/close 51731 (time 1763081632.15 total 634.13 last 86.31) - open/close 52642 (time 1763081642.16 total 644.14 last 90.94) - open/close 53460 (time 1763081652.17 total 654.15 last 81.74) - open/close 54292 (time 1763081662.18 total 664.16 last 83.17) - open/close 55024 (time 1763081672.19 total 674.17 last 73.10) - open/close 55705 (time 1763081682.19 total 684.17 last 68.08) - open/close 56445 (time 1763081692.20 total 694.18 last 73.98) - open/close 57163 (time 1763081702.22 total 704.20 last 71.65) - open/close 57991 (time 1763081712.22 total 714.20 last 82.75) - open/close 58869 (time 1763081722.23 total 724.21 last 87.75) - open/close 59756 (time 1763081732.23 total 734.21 last 88.65) - open/close 60000 (time 1763081735.29 total 737.27 last 79.92) - open/close 60900 (time 1763081745.30 total 747.28 last 89.92) - open/close 61734 (time 1763081755.30 total 757.28 last 83.35) - open/close 62564 (time 1763081765.31 total 767.29 last 82.96) - open/close 63452 (time 1763081775.31 total 777.29 last 88.76) - open/close 64293 (time 1763081785.32 total 787.30 last 84.00) total: 64768 open/close in 793.21 seconds: 81.65 ops/second - /unlink 824 (time 1763081805.29 total 10.01 last 82.34) - /unlink 1666 (time 1763081815.29 total 20.01 last 84.18) - /unlink 2482 (time 1763081825.30 total 30.01 last 81.57) - /unlink 3313 (time 1763081835.31 total 40.03 last 83.00) - /unlink 4117 (time 1763081845.31 total 50.03 last 80.36) - /unlink 4889 (time 1763081855.32 total 60.04 last 77.13) - /unlink 5644 (time 1763081865.33 total 70.04 last 75.47) - /unlink 6465 (time 1763081875.33 total 80.05 last 82.04) - /unlink 7263 (time 1763081885.34 total 90.06 last 79.77) - /unlink 8095 (time 1763081895.34 total 100.06 last 83.16) - /unlink 8996 (time 1763081905.35 total 110.06 last 90.07) - /unlink 9857 (time 1763081915.35 total 120.07 last 86.04) - /unlink 10643 (time 1763081925.37 total 130.09 last 78.49) - /unlink 11521 (time 1763081935.38 total 140.09 last 87.74) - /unlink 12347 (time 1763081945.38 total 150.09 last 82.58) - /unlink 13196 (time 1763081955.39 total 160.10 last 84.83) - /unlink 14057 (time 1763081965.39 total 170.11 last 86.03) - /unlink 14891 (time 1763081975.42 total 180.13 last 83.20) - /unlink 15673 (time 1763081985.42 total 190.14 last 78.16) - /unlink 16533 (time 1763081995.43 total 200.14 last 85.98) - /unlink 17373 (time 1763082005.43 total 210.15 last 83.97) - /unlink 18278 (time 1763082015.44 total 220.16 last 90.40) - /unlink 19183 (time 1763082025.44 total 230.16 last 90.47) - /unlink 20000 (time 1763082034.95 total 239.66 last 85.96) - /unlink 20907 (time 1763082044.95 total 249.67 last 90.68) - /unlink 21804 (time 1763082054.96 total 259.68 last 89.61) - /unlink 22676 (time 1763082064.96 total 269.68 last 87.19) - /unlink 23615 (time 1763082074.96 total 279.68 last 93.88) - /unlink 24417 (time 1763082084.97 total 289.68 last 80.17) - /unlink 25359 (time 1763082094.97 total 299.69 last 94.18) - /unlink 26186 (time 1763082104.97 total 309.69 last 82.69) - /unlink 27069 (time 1763082114.97 total 319.69 last 88.28) - /unlink 27891 (time 1763082124.97 total 329.69 last 82.19) - /unlink 28673 (time 1763082134.98 total 339.70 last 78.12) - /unlink 29569 (time 1763082145.00 total 349.71 last 89.49) - /unlink 30000 (time 1763082150.23 total 354.94 last 82.39) - /unlink 30847 (time 1763082160.24 total 364.95 last 84.62) - /unlink 31741 (time 1763082170.24 total 374.96 last 89.34) - /unlink 32617 (time 1763082180.25 total 384.97 last 87.52) - /unlink 33464 (time 1763082190.26 total 394.98 last 84.64) - /unlink 34379 (time 1763082200.27 total 404.99 last 91.40) - /unlink 35254 (time 1763082210.29 total 415.00 last 87.36) - /unlink 36055 (time 1763082220.29 total 425.01 last 80.05) - /unlink 36958 (time 1763082230.30 total 435.02 last 90.24) - /unlink 37728 (time 1763082240.31 total 445.03 last 76.90) - /unlink 38599 (time 1763082250.32 total 455.04 last 87.01) - /unlink 39438 (time 1763082260.32 total 465.04 last 83.90) - /unlink 40000 (time 1763082267.77 total 472.49 last 75.45) - /unlink 40676 (time 1763082277.78 total 482.50 last 67.55) - /unlink 41292 (time 1763082287.79 total 492.50 last 61.55) - /unlink 42085 (time 1763082297.79 total 502.51 last 79.27) - /unlink 42932 (time 1763082307.80 total 512.51 last 84.66) - /unlink 43786 (time 1763082317.80 total 522.52 last 85.38) - /unlink 44655 (time 1763082327.80 total 532.52 last 86.85) - /unlink 45521 (time 1763082337.81 total 542.53 last 86.56) - /unlink 46424 (time 1763082347.81 total 552.53 last 90.24) - /unlink 47356 (time 1763082357.82 total 562.54 last 93.14) - /unlink 48251 (time 1763082367.82 total 572.54 last 89.47) - /unlink 49189 (time 1763082377.83 total 582.54 last 93.78) - /unlink 50000 (time 1763082386.87 total 591.59 last 89.64) - /unlink 50863 (time 1763082396.89 total 601.60 last 86.19) - /unlink 51740 (time 1763082406.89 total 611.60 last 87.68) - /unlink 52630 (time 1763082416.89 total 621.61 last 88.98) - /unlink 53533 (time 1763082426.90 total 631.61 last 90.24) - /unlink 54333 (time 1763082436.90 total 641.62 last 79.95) - /unlink 55111 (time 1763082446.91 total 651.63 last 77.72) - /unlink 55938 (time 1763082456.92 total 661.63 last 82.68) - /unlink 56793 (time 1763082466.92 total 671.64 last 85.46) - /unlink 57707 (time 1763082476.92 total 681.64 last 91.37) - /unlink 58570 (time 1763082486.93 total 691.65 last 86.24) - /unlink 59492 (time 1763082496.94 total 701.65 last 92.14) - /unlink 60000 (time 1763082502.93 total 707.65 last 84.77) - /unlink 60849 (time 1763082512.93 total 717.65 last 84.89) - /unlink 61702 (time 1763082522.94 total 727.66 last 85.23) - /unlink 62568 (time 1763082532.95 total 737.66 last 86.53) - /unlink 63444 (time 1763082542.96 total 747.68 last 87.46) - /unlink 64318 (time 1763082552.97 total 757.69 last 87.35) total: 64768 /unlink in 763.41 seconds: 84.84 ops/second Waiting for MDT destroys to complete oleg437-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 - open/close 802 (time 1763082582.57 total 10.01 last 80.15) - open/close 1661 (time 1763082592.58 total 20.01 last 85.86) - open/close 2485 (time 1763082602.59 total 30.02 last 82.33) - open/close 3331 (time 1763082612.59 total 40.02 last 84.56) - open/close 4113 (time 1763082622.60 total 50.03 last 78.14) - open/close 4937 (time 1763082632.61 total 60.04 last 82.34) - open/close 5696 (time 1763082642.61 total 70.05 last 75.84) - open/close 6507 (time 1763082652.62 total 80.05 last 81.05) - open/close 7320 (time 1763082662.63 total 90.06 last 81.20) - open/close 8102 (time 1763082672.64 total 100.07 last 78.13) - open/close 8893 (time 1763082682.65 total 110.08 last 79.06) - open/close 9649 (time 1763082692.66 total 120.09 last 75.48) - open/close 10000 (time 1763082696.96 total 124.40 last 81.59) - open/close 10782 (time 1763082706.97 total 134.40 last 78.18) - open/close 11604 (time 1763082716.99 total 144.42 last 82.04) - open/close 12440 (time 1763082726.99 total 154.42 last 83.59) - open/close 13290 (time 1763082736.99 total 164.42 last 84.97) - open/close 14057 (time 1763082746.99 total 174.42 last 76.70) - open/close 14944 (time 1763082757.00 total 184.43 last 88.63) - open/close 15816 (time 1763082767.00 total 194.44 last 87.14) - open/close 16723 (time 1763082777.01 total 204.44 last 90.68) - open/close 17646 (time 1763082787.02 total 214.45 last 92.19) - open/close 18500 (time 1763082797.02 total 224.46 last 85.35) - open/close 19342 (time 1763082807.03 total 234.47 last 84.12) - open/close 20000 (time 1763082814.46 total 241.89 last 88.63) - open/close 20817 (time 1763082824.46 total 251.90 last 81.65) - open/close 21729 (time 1763082834.47 total 261.91 last 91.13) - open/close 22493 (time 1763082844.47 total 271.91 last 76.39) - open/close 23320 (time 1763082854.48 total 281.92 last 82.61) - open/close 24097 (time 1763082864.49 total 291.92 last 77.66) - open/close 24775 (time 1763082874.49 total 301.92 last 67.79) - open/close 25532 (time 1763082884.50 total 311.93 last 75.64) - open/close 26340 (time 1763082894.51 total 321.95 last 80.68) - open/close 27118 (time 1763082904.52 total 331.95 last 77.76) - open/close 27852 (time 1763082914.52 total 341.95 last 73.38) - open/close 28671 (time 1763082924.53 total 351.96 last 81.84) - open/close 29470 (time 1763082934.53 total 361.97 last 79.86) - open/close 30000 (time 1763082940.83 total 368.26 last 84.22) - open/close 30875 (time 1763082950.84 total 378.27 last 87.41) - open/close 31627 (time 1763082960.84 total 388.28 last 75.16) - open/close 32392 (time 1763082970.85 total 398.28 last 76.47) - open/close 33282 (time 1763082980.86 total 408.30 last 88.84) - open/close 34129 (time 1763082990.87 total 418.30 last 84.69) - open/close 35077 (time 1763083000.87 total 428.30 last 94.78) - open/close 36030 (time 1763083010.88 total 438.31 last 95.20) - open/close 36933 (time 1763083020.88 total 448.31 last 90.29) - open/close 37830 (time 1763083030.88 total 458.31 last 89.69) - open/close 38590 (time 1763083040.88 total 468.32 last 75.99) - open/close 39409 (time 1763083050.89 total 478.32 last 81.86) - open/close 40000 (time 1763083057.90 total 485.34 last 84.23) - open/close 40911 (time 1763083067.91 total 495.34 last 91.07) - open/close 41773 (time 1763083077.92 total 505.35 last 86.10) - open/close 42640 (time 1763083087.93 total 515.36 last 86.64) - open/close 43464 (time 1763083097.94 total 525.37 last 82.32) - open/close 44336 (time 1763083107.94 total 535.37 last 87.19) - open/close 45102 (time 1763083117.94 total 545.38 last 76.55) - open/close 45940 (time 1763083127.94 total 555.38 last 83.79) - open/close 46744 (time 1763083137.95 total 565.39 last 80.32) - open/close 47595 (time 1763083147.96 total 575.40 last 85.03) - open/close 48427 (time 1763083157.97 total 585.40 last 83.15) - open/close 49306 (time 1763083167.97 total 595.40 last 87.88) - open/close 50000 (time 1763083175.57 total 603.00 last 91.32) - open/close 50808 (time 1763083185.57 total 613.01 last 80.77) - open/close 51574 (time 1763083195.59 total 623.02 last 76.50) - open/close 52336 (time 1763083205.59 total 633.03 last 76.15) - open/close 53112 (time 1763083215.60 total 643.04 last 77.54) - open/close 53912 (time 1763083225.60 total 653.04 last 79.99) - open/close 54728 (time 1763083235.61 total 663.04 last 81.55) - open/close 55601 (time 1763083245.61 total 673.04 last 87.30) - open/close 56524 (time 1763083255.61 total 683.05 last 92.28) - open/close 57446 (time 1763083265.62 total 693.05 last 92.12) - open/close 58349 (time 1763083275.62 total 703.06 last 90.27) - open/close 59225 (time 1763083285.63 total 713.07 last 87.53) - open/close 60000 (time 1763083294.26 total 721.69 last 89.84) - open/close 60795 (time 1763083304.26 total 731.70 last 79.46) - open/close 61668 (time 1763083314.27 total 741.70 last 87.25) - open/close 62523 (time 1763083324.28 total 751.71 last 85.45) - open/close 63346 (time 1763083334.28 total 761.71 last 82.29) - open/close 64261 (time 1763083344.28 total 771.71 last 91.48) total: 64768 open/close in 777.45 seconds: 83.31 ops/second - /unlink 959 (time 1763083363.09 total 10.01 last 95.84) - /unlink 1792 (time 1763083373.12 total 20.03 last 83.08) - /unlink 2672 (time 1763083383.12 total 30.04 last 87.95) - /unlink 3540 (time 1763083393.13 total 40.04 last 86.78) - /unlink 4383 (time 1763083403.13 total 50.04 last 84.26) - /unlink 5280 (time 1763083413.13 total 60.05 last 89.66) - /unlink 6226 (time 1763083423.14 total 70.05 last 94.56) - /unlink 7085 (time 1763083433.15 total 80.06 last 85.80) - /unlink 7897 (time 1763083443.15 total 90.07 last 81.18) - /unlink 8767 (time 1763083453.16 total 100.07 last 86.93) - /unlink 9663 (time 1763083463.17 total 110.09 last 89.49) - /unlink 10000 (time 1763083466.83 total 113.74 last 92.19) - /unlink 10868 (time 1763083476.83 total 123.75 last 86.77) - /unlink 11693 (time 1763083486.83 total 133.75 last 82.48) - /unlink 12557 (time 1763083496.84 total 143.76 last 86.31) - /unlink 13424 (time 1763083506.85 total 153.76 last 86.66) - /unlink 14257 (time 1763083516.86 total 163.78 last 83.16) - /unlink 15115 (time 1763083526.87 total 173.78 last 85.79) - /unlink 15967 (time 1763083536.87 total 183.78 last 85.16) - /unlink 16850 (time 1763083546.88 total 193.79 last 88.24) - /unlink 17715 (time 1763083556.89 total 203.80 last 86.43) - /unlink 18583 (time 1763083566.89 total 213.81 last 86.74) - /unlink 19440 (time 1763083576.91 total 223.82 last 85.58) - /unlink 20000 (time 1763083582.93 total 229.84 last 92.98) - /unlink 20852 (time 1763083592.94 total 239.85 last 85.14) - /unlink 21778 (time 1763083602.94 total 249.85 last 92.58) - /unlink 22813 (time 1763083612.94 total 259.86 last 103.44) - /unlink 23750 (time 1763083622.95 total 269.87 last 93.63) - /unlink 24676 (time 1763083632.96 total 279.88 last 92.51) - /unlink 25632 (time 1763083642.97 total 289.88 last 95.53) - /unlink 26640 (time 1763083652.97 total 299.88 last 100.79) - /unlink 27577 (time 1763083662.97 total 309.88 last 93.69) - /unlink 28501 (time 1763083672.97 total 319.89 last 92.36) - /unlink 29497 (time 1763083682.98 total 329.90 last 99.52) - /unlink 30000 (time 1763083688.22 total 335.13 last 96.05) - /unlink 30917 (time 1763083698.22 total 345.14 last 91.68) - /unlink 31775 (time 1763083708.23 total 355.15 last 85.73) - /unlink 32666 (time 1763083718.23 total 365.15 last 89.10) - /unlink 33593 (time 1763083728.24 total 375.16 last 92.61) - /unlink 34411 (time 1763083738.24 total 385.16 last 81.80) - /unlink 35266 (time 1763083748.25 total 395.17 last 85.39) - /unlink 36163 (time 1763083758.26 total 405.18 last 89.61) - /unlink 37069 (time 1763083768.28 total 415.19 last 90.47) - /unlink 37918 (time 1763083778.29 total 425.21 last 84.75) - /unlink 38844 (time 1763083788.30 total 435.22 last 92.52) - /unlink 39741 (time 1763083798.31 total 445.22 last 89.67) - /unlink 40000 (time 1763083801.24 total 448.16 last 88.20) - /unlink 40937 (time 1763083811.25 total 458.16 last 93.67) - /unlink 41885 (time 1763083821.25 total 468.17 last 94.74) - /unlink 42840 (time 1763083831.26 total 478.18 last 95.42) - /unlink 43761 (time 1763083841.26 total 488.18 last 92.08) - /unlink 44699 (time 1763083851.27 total 498.18 last 93.75) - /unlink 45652 (time 1763083861.27 total 508.19 last 95.24) - /unlink 46557 (time 1763083871.29 total 518.20 last 90.40) - /unlink 47421 (time 1763083881.29 total 528.20 last 86.39) - /unlink 48313 (time 1763083891.29 total 538.20 last 89.20) - /unlink 49216 (time 1763083901.30 total 548.21 last 90.22) - /unlink 50000 (time 1763083910.06 total 556.98 last 89.42) - /unlink 50970 (time 1763083920.06 total 566.98 last 97.00) - /unlink 51924 (time 1763083930.07 total 576.98 last 95.36) - /unlink 52902 (time 1763083940.08 total 586.99 last 97.69) - /unlink 53941 (time 1763083950.09 total 597.00 last 103.83) - /unlink 55003 (time 1763083960.09 total 607.00 last 106.18) - /unlink 56066 (time 1763083970.10 total 617.01 last 106.19) - /unlink 57072 (time 1763083980.10 total 627.02 last 100.53) - /unlink 58177 (time 1763083990.11 total 637.02 last 110.44) - /unlink 59384 (time 1763084000.11 total 647.03 last 120.65) - /unlink 60000 (time 1763084005.86 total 652.77 last 107.27) - /unlink 61030 (time 1763084015.86 total 662.77 last 103.00) - /unlink 62170 (time 1763084025.86 total 672.78 last 113.95) - /unlink 63360 (time 1763084035.86 total 682.78 last 118.98) - /unlink 64695 (time 1763084045.87 total 692.79 last 133.38) total: 64768 /unlink in 693.22 seconds: 93.43 ops/second Waiting for MDT destroys to complete oleg437-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 - open/close 1232 (time 1763084064.67 total 10.01 last 123.10) - open/close 2248 (time 1763084074.67 total 20.01 last 101.59) - open/close 3465 (time 1763084084.68 total 30.02 last 121.63) - open/close 4625 (time 1763084094.69 total 40.02 last 115.92) - open/close 5631 (time 1763084104.69 total 50.03 last 100.56) - open/close 6619 (time 1763084114.69 total 60.03 last 98.77) - open/close 7710 (time 1763084124.69 total 70.03 last 109.10) - open/close 8841 (time 1763084134.70 total 80.03 last 113.08) - open/close 10000 (time 1763084144.34 total 89.67 last 120.22) - open/close 11082 (time 1763084154.34 total 99.68 last 108.15) - open/close 12340 (time 1763084164.34 total 109.68 last 125.80) - open/close 13599 (time 1763084174.36 total 119.70 last 125.67) - open/close 14758 (time 1763084184.36 total 129.70 last 115.88) - open/close 16048 (time 1763084194.36 total 139.70 last 128.97) - open/close 17093 (time 1763084204.36 total 149.70 last 104.49) - open/close 18368 (time 1763084214.37 total 159.71 last 127.41) - open/close 19611 (time 1763084224.37 total 169.71 last 124.26) - open/close 20000 (time 1763084227.94 total 173.27 last 109.25) - open/close 21161 (time 1763084237.95 total 183.28 last 115.96) - open/close 22253 (time 1763084247.96 total 193.29 last 109.11) - open/close 23684 (time 1763084257.96 total 203.29 last 143.10) - open/close 25192 (time 1763084267.96 total 213.29 last 150.77) - open/close 26326 (time 1763084277.96 total 223.30 last 113.36) - open/close 27448 (time 1763084287.96 total 233.30 last 112.17) - open/close 28359 (time 1763084297.97 total 243.30 last 91.05) - open/close 29375 (time 1763084307.98 total 253.31 last 101.52) - open/close 30000 (time 1763084313.51 total 258.85 last 112.91) - open/close 31017 (time 1763084323.51 total 268.85 last 101.69) - open/close 31978 (time 1763084333.52 total 278.85 last 96.06) - open/close 33195 (time 1763084343.52 total 288.86 last 121.63) - open/close 34184 (time 1763084353.53 total 298.86 last 98.88) - open/close 35120 (time 1763084363.54 total 308.87 last 93.48) - open/close 36391 (time 1763084373.54 total 318.88 last 127.06) - open/close 37535 (time 1763084383.55 total 328.89 last 114.30) - open/close 38567 (time 1763084393.56 total 338.89 last 103.11) - open/close 39524 (time 1763084403.56 total 348.90 last 95.67) - open/close 40000 (time 1763084407.68 total 353.01 last 115.65) - open/close 41076 (time 1763084417.68 total 363.02 last 107.58) - open/close 42184 (time 1763084427.68 total 373.02 last 110.78) - open/close 43576 (time 1763084437.69 total 383.02 last 139.15) - open/close 44698 (time 1763084447.69 total 393.02 last 112.18) - open/close 45875 (time 1763084457.69 total 403.03 last 117.63) - open/close 47436 (time 1763084467.69 total 413.03 last 156.10) - open/close 49202 (time 1763084477.70 total 423.03 last 176.52) - open/close 50000 (time 1763084481.47 total 426.81 last 211.42) - open/close 51663 (time 1763084491.48 total 436.81 last 166.22) - open/close 52844 (time 1763084501.48 total 446.82 last 118.04) - open/close 54164 (time 1763084511.48 total 456.82 last 131.97) - open/close 55939 (time 1763084521.48 total 466.82 last 177.50) - open/close 57706 (time 1763084531.49 total 476.82 last 176.62) - open/close 59116 (time 1763084541.49 total 486.83 last 140.95) - open/close 60000 (time 1763084548.20 total 493.54 last 131.69) - open/close 62318 (time 1763084558.21 total 503.55 last 231.67) - open/close 64015 (time 1763084568.21 total 513.55 last 169.67) total: 64768 open/close in 516.78 seconds: 125.33 ops/second - /unlink 5417 (time 1763084583.11 total 10.00 last 541.63) - /unlink 10000 (time 1763084590.15 total 17.05 last 650.65) - /unlink 16423 (time 1763084600.15 total 27.05 last 642.21) - /unlink 20000 (time 1763084607.51 total 34.40 last 486.12) - /unlink 25348 (time 1763084617.51 total 44.41 last 534.71) - /unlink 30000 (time 1763084627.15 total 54.05 last 482.48) - /unlink 34526 (time 1763084637.15 total 64.05 last 452.56) - /unlink 39960 (time 1763084647.16 total 74.05 last 543.31) - /unlink 44633 (time 1763084657.16 total 84.05 last 467.28) - /unlink 49464 (time 1763084667.16 total 94.05 last 483.10) - /unlink 55170 (time 1763084677.16 total 104.05 last 570.59) - /unlink 59336 (time 1763084687.16 total 114.05 last 416.58) - /unlink 64122 (time 1763084697.16 total 124.05 last 478.58) total: 64768 /unlink in 125.17 seconds: 517.46 ops/second Waiting for MDT destroys to complete oleg437-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 - open/close 1980 (time 1763084714.35 total 10.00 last 197.98) - open/close 3972 (time 1763084724.36 total 20.01 last 199.11) - open/close 5926 (time 1763084734.36 total 30.01 last 195.37) - open/close 7990 (time 1763084744.36 total 40.01 last 206.31) - open/close 10000 (time 1763084753.20 total 48.85 last 227.33) - open/close 12508 (time 1763084763.21 total 58.86 last 250.72) - open/close 14731 (time 1763084773.21 total 68.86 last 222.19) - open/close 17007 (time 1763084783.21 total 78.86 last 227.56) - open/close 18926 (time 1763084793.22 total 88.87 last 191.84) - open/close 20000 (time 1763084798.44 total 94.09 last 205.48) - open/close 22297 (time 1763084808.44 total 104.09 last 229.66) - open/close 24391 (time 1763084818.45 total 114.10 last 209.36) - open/close 26399 (time 1763084828.45 total 124.10 last 200.72) - open/close 28513 (time 1763084838.45 total 134.10 last 211.35) - open/close 30000 (time 1763084844.39 total 140.04 last 250.44) - open/close 32364 (time 1763084854.39 total 150.04 last 236.34) - open/close 35153 (time 1763084864.40 total 160.05 last 278.84) - open/close 37963 (time 1763084874.40 total 170.05 last 280.94) - open/close 40000 (time 1763084882.58 total 178.23 last 248.90) - open/close 41984 (time 1763084892.58 total 188.23 last 198.34) - open/close 44358 (time 1763084902.58 total 198.23 last 237.40) - open/close 46792 (time 1763084912.59 total 208.24 last 243.31) - open/close 48838 (time 1763084922.59 total 218.24 last 204.58) - open/close 50000 (time 1763084927.29 total 222.94 last 247.16) - open/close 52437 (time 1763084937.29 total 232.94 last 243.64) - open/close 55457 (time 1763084947.29 total 242.94 last 301.98) - open/close 58947 (time 1763084957.29 total 252.94 last 348.96) - open/close 60000 (time 1763084960.24 total 255.89 last 357.73) - open/close 62740 (time 1763084970.24 total 265.89 last 273.94) total: 64768 open/close in 272.41 seconds: 237.76 ops/second - /unlink 5772 (time 1763084988.31 total 10.00 last 577.14) - /unlink 10000 (time 1763084997.33 total 19.02 last 468.76) - /unlink 14305 (time 1763085007.33 total 29.02 last 430.49) - /unlink 19984 (time 1763085017.33 total 39.02 last 567.85) - /unlink 25728 (time 1763085027.33 total 49.02 last 574.36) - /unlink 30000 (time 1763085035.10 total 56.80 last 549.55) - /unlink 35110 (time 1763085045.10 total 66.80 last 510.93) - /unlink 40000 (time 1763085053.36 total 75.05 last 592.48) - /unlink 45431 (time 1763085063.36 total 85.05 last 543.03) - /unlink 50000 (time 1763085072.05 total 93.74 last 525.76) - /unlink 55788 (time 1763085082.05 total 103.74 last 578.77) - /unlink 60000 (time 1763085089.85 total 111.54 last 539.93) total: 64768 /unlink in 119.84 seconds: 540.46 ops/second Waiting for MDT destroys to complete oleg437-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 - open/close 3426 (time 1763085118.66 total 10.00 last 342.58) - open/close 6180 (time 1763085128.67 total 20.00 last 275.33) - open/close 8200 (time 1763085138.67 total 30.01 last 201.94) - open/close 10000 (time 1763085146.27 total 37.60 last 236.94) - open/close 12780 (time 1763085156.27 total 47.61 last 277.86) - open/close 14658 (time 1763085166.28 total 57.61 last 187.70) - open/close 17327 (time 1763085176.28 total 67.61 last 266.85) - open/close 19878 (time 1763085186.28 total 77.62 last 255.07) - open/close 22202 (time 1763085196.28 total 87.62 last 232.28) - open/close 24276 (time 1763085206.29 total 97.62 last 207.33) - open/close 26584 (time 1763085216.29 total 107.63 last 230.71) - open/close 28868 (time 1763085226.29 total 117.63 last 228.36) - open/close 30000 (time 1763085231.86 total 123.20 last 203.28) - open/close 32404 (time 1763085241.86 total 133.20 last 240.35) - open/close 34868 (time 1763085251.86 total 143.20 last 246.39) - open/close 37630 (time 1763085261.87 total 153.20 last 276.16) - open/close 40000 (time 1763085270.99 total 162.33 last 259.71) - open/close 43197 (time 1763085280.99 total 172.33 last 319.69) - open/close 45989 (time 1763085290.99 total 182.33 last 279.18) - open/close 48687 (time 1763085300.99 total 192.33 last 269.77) - open/close 50000 (time 1763085306.44 total 197.78 last 240.99) - open/close 52858 (time 1763085316.45 total 207.78 last 285.71) - open/close 55417 (time 1763085326.45 total 217.78 last 255.86) - open/close 57509 (time 1763085336.45 total 227.79 last 209.15) - open/close 59270 (time 1763085346.45 total 237.79 last 176.06) - open/close 60000 (time 1763085350.60 total 241.94 last 176.00) - open/close 61893 (time 1763085360.60 total 251.94 last 189.24) - open/close 63760 (time 1763085370.60 total 261.94 last 186.68) total: 64768 open/close in 268.14 seconds: 241.55 ops/second - /unlink 3775 (time 1763085388.56 total 10.00 last 377.48) - /unlink 7417 (time 1763085398.56 total 20.00 last 364.17) - /unlink 10000 (time 1763085404.82 total 26.26 last 412.63) - /unlink 14065 (time 1763085414.82 total 36.26 last 406.47) - /unlink 17718 (time 1763085424.82 total 46.26 last 365.26) - /unlink 20000 (time 1763085429.42 total 50.87 last 495.54) - /unlink 25849 (time 1763085439.42 total 60.87 last 584.87) - /unlink 29938 (time 1763085449.42 total 70.87 last 408.88) - /unlink 34741 (time 1763085459.42 total 80.87 last 480.29) - /unlink 40000 (time 1763085469.03 total 90.47 last 547.60) - /unlink 44093 (time 1763085479.03 total 100.48 last 409.20) - /unlink 48268 (time 1763085489.03 total 110.48 last 417.49) - /unlink 50000 (time 1763085492.91 total 114.35 last 446.84) - /unlink 53601 (time 1763085502.91 total 124.35 last 360.00) - /unlink 57857 (time 1763085512.91 total 134.36 last 425.56) - /unlink 60000 (time 1763085517.59 total 139.03 last 458.38) total: 64768 /unlink in 146.07 seconds: 443.40 ops/second Waiting for MDT destroys to complete oleg437-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg437-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local oleg437-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg437-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 PASS 106 (4679s) Stopping clients: oleg437-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg437-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 oleg437-server: oleg437-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4918 sec ========== 20:59:47 (1763085587) === conf-sanity: start cleanup 20:59:47 (1763085587) === === conf-sanity: finish cleanup 20:59:49 (1763085589) ===