-----============= acceptance-small: conf-sanity ============----- Fri Nov 14 15:56:27 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg149-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball Stopping clients: oleg149-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg149-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg149-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg149-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg149-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg149-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg149-server oleg149-server: oleg149-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg149-server' oleg149-server: oleg149-server.virtnet: executing load_modules_local oleg149-server: Loading modules from /home/green/git/lustre-release/lustre oleg149-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg149-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg149-server: detected 4 online CPUs by sysfs oleg149-server: MODOPTS_LIBCFS= oleg149-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg149-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg149-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg149-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg149-server: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg149-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg149-server: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg149-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg149-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg149-server stop mds service on oleg149-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg149-server stop mds service on oleg149-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg149-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:59:14 (1763153954) Stopping clients: oleg149-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg149-client.virtnet /mnt/lustre2 (opts:-f) oleg149-server: oleg149-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg149-server' oleg149-server: oleg149-server.virtnet: executing load_modules_local oleg149-server: Loading modules from /home/green/git/lustre-release/lustre oleg149-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg149-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg149-server: detected 4 online CPUs by sysfs oleg149-server: MODOPTS_LIBCFS= oleg149-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg149-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg149-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg149-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg149-client.virtnet: -o user_xattr,flock 192.168.201.149@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 764 (time 1763154060.02 total 10.00 last 76.39) - open/close 1435 (time 1763154070.02 total 20.00 last 67.09) - open/close 2214 (time 1763154080.03 total 30.01 last 77.86) - open/close 2989 (time 1763154090.04 total 40.02 last 77.40) - open/close 3757 (time 1763154100.04 total 50.02 last 76.79) - open/close 4514 (time 1763154110.05 total 60.03 last 75.67) - open/close 5263 (time 1763154120.06 total 70.04 last 74.81) - open/close 6005 (time 1763154130.06 total 80.04 last 74.18) - open/close 6731 (time 1763154140.07 total 90.05 last 72.53) - open/close 7513 (time 1763154150.08 total 100.05 last 78.17) - open/close 8281 (time 1763154160.08 total 110.06 last 76.74) - open/close 8995 (time 1763154170.09 total 120.07 last 71.35) - open/close 9822 (time 1763154180.10 total 130.07 last 82.65) - open/close 10000 (time 1763154182.28 total 132.26 last 81.42) - open/close 10818 (time 1763154192.29 total 142.27 last 81.76) - open/close 11608 (time 1763154202.31 total 152.28 last 78.86) - open/close 12350 (time 1763154212.31 total 162.29 last 74.15) - open/close 13050 (time 1763154222.31 total 172.29 last 69.99) - open/close 13758 (time 1763154232.32 total 182.30 last 70.72) - open/close 14474 (time 1763154242.34 total 192.32 last 71.46) - open/close 15166 (time 1763154252.35 total 202.32 last 69.19) - open/close 15875 (time 1763154262.35 total 212.33 last 70.84) - open/close 16588 (time 1763154272.36 total 222.34 last 71.24) - open/close 17372 (time 1763154282.37 total 232.34 last 78.38) - open/close 18140 (time 1763154292.37 total 242.35 last 76.73) - open/close 18851 (time 1763154302.39 total 252.37 last 71.00) - open/close 19556 (time 1763154312.41 total 262.38 last 70.38) - open/close 20000 (time 1763154318.44 total 268.42 last 73.52) - open/close 20744 (time 1763154328.45 total 278.43 last 74.33) - open/close 21482 (time 1763154338.47 total 288.44 last 73.70) - open/close 22075 (time 1763154348.48 total 298.46 last 59.22) - open/close 22743 (time 1763154358.50 total 308.48 last 66.68) - open/close 23380 (time 1763154368.51 total 318.49 last 63.64) - open/close 24051 (time 1763154378.51 total 328.49 last 67.10) - open/close 24758 (time 1763154388.52 total 338.50 last 70.63) - open/close 25532 (time 1763154398.53 total 348.51 last 77.31) - open/close 26226 (time 1763154408.54 total 358.52 last 69.34) - open/close 26931 (time 1763154418.54 total 368.52 last 70.46) - open/close 27600 (time 1763154428.56 total 378.54 last 66.79) - open/close 28379 (time 1763154438.56 total 388.54 last 77.85) - open/close 29138 (time 1763154448.57 total 398.55 last 75.83) - open/close 29886 (time 1763154458.58 total 408.56 last 74.76) - open/close 30634 (time 1763154468.59 total 418.57 last 74.72) - open/close 31371 (time 1763154478.60 total 428.57 last 73.66) - open/close 32146 (time 1763154488.60 total 438.58 last 77.44) - open/close 32895 (time 1763154498.60 total 448.58 last 74.90) - open/close 33654 (time 1763154508.62 total 458.60 last 75.80) - open/close 34412 (time 1763154518.62 total 468.60 last 75.78) - open/close 35161 (time 1763154528.63 total 478.61 last 74.83) - open/close 35898 (time 1763154538.63 total 488.61 last 73.68) - open/close 36631 (time 1763154548.64 total 498.62 last 73.24) - open/close 37403 (time 1763154558.64 total 508.62 last 77.19) - open/close 38174 (time 1763154568.65 total 518.62 last 77.07) - open/close 38955 (time 1763154578.65 total 528.63 last 78.03) - open/close 39678 (time 1763154588.66 total 538.64 last 72.24) - open/close 40000 (time 1763154593.60 total 543.57 last 65.27) - open/close 40740 (time 1763154603.61 total 553.59 last 73.87) - open/close 41486 (time 1763154613.61 total 563.59 last 74.59) - open/close 42255 (time 1763154623.63 total 573.61 last 76.76) - open/close 42977 (time 1763154633.65 total 583.63 last 72.09) - open/close 43754 (time 1763154643.67 total 593.65 last 77.50) - open/close 44533 (time 1763154653.68 total 603.66 last 77.87) - open/close 45275 (time 1763154663.70 total 613.67 last 74.07) - open/close 46057 (time 1763154673.70 total 623.68 last 78.16) - open/close 46853 (time 1763154683.71 total 633.69 last 79.55) - open/close 47581 (time 1763154693.71 total 643.69 last 72.80) - open/close 48339 (time 1763154703.72 total 653.70 last 75.73) - open/close 49085 (time 1763154713.72 total 663.70 last 74.55) - open/close 49810 (time 1763154723.73 total 673.71 last 72.48) - open/close 50000 (time 1763154726.46 total 676.44 last 69.46) - open/close 50773 (time 1763154736.47 total 686.45 last 77.26) - open/close 51522 (time 1763154746.48 total 696.46 last 74.82) - open/close 52282 (time 1763154756.48 total 706.46 last 75.98) - open/close 53025 (time 1763154766.48 total 716.46 last 74.29) - open/close 53754 (time 1763154776.49 total 726.47 last 72.83) - open/close 54496 (time 1763154786.50 total 736.48 last 74.14) - open/close 55258 (time 1763154796.51 total 746.49 last 76.12) - open/close 55965 (time 1763154806.52 total 756.50 last 70.65) - open/close 56679 (time 1763154816.52 total 766.50 last 71.39) - open/close 57391 (time 1763154826.53 total 776.50 last 71.16) - open/close 58143 (time 1763154836.53 total 786.51 last 75.19) - open/close 58877 (time 1763154846.53 total 796.51 last 73.38) - open/close 59601 (time 1763154856.54 total 806.52 last 72.33) - open/close 60000 (time 1763154862.52 total 812.50 last 66.69) - open/close 60720 (time 1763154872.54 total 822.52 last 71.87) - open/close 61429 (time 1763154882.55 total 832.53 last 70.85) - open/close 62173 (time 1763154892.55 total 842.53 last 74.39) - open/close 62856 (time 1763154902.56 total 852.54 last 68.22) - open/close 63569 (time 1763154912.56 total 862.54 last 71.29) - open/close 64231 (time 1763154922.57 total 872.55 last 66.15) total: 64768 open/close in 880.24 seconds: 73.58 ops/second - /unlink 781 (time 1763154943.61 total 10.00 last 78.07) - /unlink 1506 (time 1763154953.61 total 20.01 last 72.47) - /unlink 2197 (time 1763154963.62 total 30.02 last 68.99) - /unlink 2912 (time 1763154973.63 total 40.03 last 71.47) - /unlink 3580 (time 1763154983.65 total 50.04 last 66.69) - /unlink 4295 (time 1763154993.66 total 60.06 last 71.41) - /unlink 5020 (time 1763155003.66 total 70.06 last 72.45) - /unlink 5646 (time 1763155013.66 total 80.06 last 62.60) - /unlink 6294 (time 1763155023.67 total 90.07 last 64.76) - /unlink 7017 (time 1763155033.68 total 100.08 last 72.21) - /unlink 7698 (time 1763155043.71 total 110.11 last 67.92) - /unlink 8378 (time 1763155053.72 total 120.11 last 67.95) - /unlink 9089 (time 1763155063.72 total 130.12 last 71.09) - /unlink 9810 (time 1763155073.73 total 140.12 last 72.04) - /unlink 10000 (time 1763155077.26 total 143.66 last 53.78) - /unlink 10661 (time 1763155087.28 total 153.67 last 65.99) - /unlink 11231 (time 1763155097.28 total 163.68 last 56.96) - /unlink 11943 (time 1763155107.29 total 173.69 last 71.15) - /unlink 12674 (time 1763155117.31 total 183.70 last 72.99) - /unlink 13412 (time 1763155127.32 total 193.72 last 73.71) - /unlink 14141 (time 1763155137.33 total 203.73 last 72.79) - /unlink 14900 (time 1763155147.35 total 213.75 last 75.76) - /unlink 15647 (time 1763155157.37 total 223.77 last 74.58) - /unlink 16377 (time 1763155167.38 total 233.77 last 72.94) - /unlink 17046 (time 1763155177.40 total 243.80 last 66.75) - /unlink 17736 (time 1763155187.41 total 253.81 last 68.91) - /unlink 18443 (time 1763155197.41 total 263.81 last 70.68) - /unlink 19141 (time 1763155207.42 total 273.81 last 69.80) - /unlink 19862 (time 1763155217.42 total 283.82 last 72.08) - /unlink 20567 (time 1763155227.44 total 293.83 last 70.37) - /unlink 21274 (time 1763155237.44 total 303.84 last 70.64) - /unlink 21954 (time 1763155247.46 total 313.86 last 67.89) - /unlink 22692 (time 1763155257.46 total 323.86 last 73.77) - /unlink 23372 (time 1763155267.47 total 333.86 last 67.98) - /unlink 24093 (time 1763155277.47 total 343.87 last 72.07) - /unlink 24824 (time 1763155287.48 total 353.88 last 73.00) - /unlink 25588 (time 1763155297.49 total 363.89 last 76.34) - /unlink 26198 (time 1763155307.50 total 373.90 last 60.93) - /unlink 26862 (time 1763155317.51 total 383.91 last 66.33) - /unlink 27590 (time 1763155327.54 total 393.94 last 72.62) - /unlink 28345 (time 1763155337.54 total 403.94 last 75.45) - /unlink 28999 (time 1763155347.55 total 413.95 last 65.39) - /unlink 29696 (time 1763155357.55 total 423.95 last 69.70) - /unlink 30000 (time 1763155361.83 total 428.23 last 71.02) - /unlink 30693 (time 1763155371.83 total 438.23 last 69.30) - /unlink 31399 (time 1763155381.83 total 448.23 last 70.58) - /unlink 32139 (time 1763155391.83 total 458.23 last 73.97) - /unlink 32876 (time 1763155401.87 total 468.27 last 73.42) - /unlink 33580 (time 1763155411.88 total 478.28 last 70.36) - /unlink 34303 (time 1763155421.89 total 488.29 last 72.23) - /unlink 35091 (time 1763155431.89 total 498.29 last 78.77) - /unlink 35834 (time 1763155441.90 total 508.30 last 74.23) - /unlink 36549 (time 1763155451.91 total 518.31 last 71.43) - /unlink 37229 (time 1763155461.93 total 528.32 last 67.89) - /unlink 37941 (time 1763155471.93 total 538.33 last 71.19) - /unlink 38654 (time 1763155481.93 total 548.33 last 71.29) - /unlink 39397 (time 1763155491.93 total 558.33 last 74.26) - /unlink 40000 (time 1763155500.94 total 567.33 last 66.98) - /unlink 40667 (time 1763155510.94 total 577.33 last 66.70) - /unlink 41370 (time 1763155520.94 total 587.34 last 70.29) - /unlink 42057 (time 1763155530.94 total 597.34 last 68.69) - /unlink 42773 (time 1763155540.94 total 607.34 last 71.56) - /unlink 43504 (time 1763155550.95 total 617.35 last 73.05) - /unlink 44283 (time 1763155560.96 total 627.36 last 77.83) - /unlink 45107 (time 1763155570.97 total 637.37 last 82.29) - /unlink 45796 (time 1763155580.98 total 647.37 last 68.88) - /unlink 46566 (time 1763155590.99 total 657.39 last 76.91) - /unlink 47362 (time 1763155601.00 total 667.40 last 79.51) - /unlink 48138 (time 1763155611.01 total 677.41 last 77.52) - /unlink 48868 (time 1763155621.01 total 687.41 last 72.98) - /unlink 49667 (time 1763155631.02 total 697.42 last 79.80) - /unlink 50000 (time 1763155635.36 total 701.76 last 76.75) - /unlink 50762 (time 1763155645.37 total 711.77 last 76.12) - /unlink 51545 (time 1763155655.39 total 721.79 last 78.20) - /unlink 52316 (time 1763155665.39 total 731.79 last 77.08) - /unlink 52995 (time 1763155675.39 total 741.79 last 67.89) - /unlink 53775 (time 1763155685.40 total 751.80 last 77.90) - /unlink 54593 (time 1763155695.41 total 761.81 last 81.74) - /unlink 55380 (time 1763155705.42 total 771.82 last 78.61) - /unlink 56142 (time 1763155715.44 total 781.84 last 76.10) - /unlink 56871 (time 1763155725.45 total 791.84 last 72.84) - /unlink 57640 (time 1763155735.46 total 801.86 last 76.79) - /unlink 58394 (time 1763155745.46 total 811.86 last 75.39) - /unlink 59180 (time 1763155755.47 total 821.87 last 78.51) - /unlink 59991 (time 1763155765.48 total 831.87 last 81.08) - /unlink 60739 (time 1763155775.48 total 841.88 last 74.74) - /unlink 61436 (time 1763155785.49 total 851.88 last 69.68) - /unlink 62129 (time 1763155795.49 total 861.89 last 69.23) - /unlink 62797 (time 1763155805.50 total 871.90 last 66.78) - /unlink 63505 (time 1763155815.50 total 881.90 last 70.76) - /unlink 64249 (time 1763155825.50 total 891.90 last 74.39) total: 64768 /unlink in 899.49 seconds: 72.01 ops/second Waiting for MDT destroys to complete oleg149-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 - open/close 727 (time 1763155856.29 total 10.02 last 72.54) - open/close 1454 (time 1763155866.31 total 20.03 last 72.61) - open/close 2179 (time 1763155876.31 total 30.03 last 72.50) - open/close 2938 (time 1763155886.33 total 40.05 last 75.75) - open/close 3617 (time 1763155896.33 total 50.05 last 67.90) - open/close 4364 (time 1763155906.33 total 60.06 last 74.65) - open/close 5079 (time 1763155916.35 total 70.07 last 71.41) - open/close 5833 (time 1763155926.36 total 80.09 last 75.32) - open/close 6512 (time 1763155936.36 total 90.09 last 67.87) - open/close 7163 (time 1763155946.37 total 100.09 last 65.07) - open/close 7869 (time 1763155956.37 total 110.10 last 70.58) - open/close 8596 (time 1763155966.37 total 120.10 last 72.68) - open/close 9336 (time 1763155976.37 total 130.10 last 73.99) - open/close 10000 (time 1763155986.19 total 139.92 last 67.66) - open/close 10705 (time 1763155996.20 total 149.93 last 70.41) - open/close 11444 (time 1763156006.20 total 159.93 last 73.88) - open/close 12179 (time 1763156016.20 total 169.93 last 73.48) - open/close 12905 (time 1763156026.21 total 179.94 last 72.55) - open/close 13647 (time 1763156036.22 total 189.94 last 74.17) - open/close 14369 (time 1763156046.23 total 199.95 last 72.13) - open/close 15094 (time 1763156056.26 total 209.99 last 72.25) - open/close 15777 (time 1763156066.26 total 219.99 last 68.27) - open/close 16470 (time 1763156076.27 total 230.00 last 69.26) - open/close 17155 (time 1763156086.28 total 240.01 last 68.45) - open/close 17866 (time 1763156096.28 total 250.01 last 71.09) - open/close 18574 (time 1763156106.28 total 260.01 last 70.77) - open/close 19301 (time 1763156116.30 total 270.02 last 72.61) - open/close 20000 (time 1763156126.29 total 280.02 last 69.92) - open/close 20731 (time 1763156136.30 total 290.03 last 73.05) - open/close 21438 (time 1763156146.30 total 300.03 last 70.69) - open/close 22176 (time 1763156156.32 total 310.05 last 73.67) - open/close 22887 (time 1763156166.33 total 320.06 last 71.05) - open/close 23647 (time 1763156176.33 total 330.06 last 75.98) - open/close 24391 (time 1763156186.33 total 340.06 last 74.39) - open/close 25119 (time 1763156196.34 total 350.07 last 72.73) - open/close 25890 (time 1763156206.34 total 360.07 last 77.07) - open/close 26639 (time 1763156216.36 total 370.09 last 74.81) - open/close 27469 (time 1763156226.36 total 380.09 last 82.95) - open/close 28270 (time 1763156236.38 total 390.10 last 80.00) - open/close 29015 (time 1763156246.39 total 400.11 last 74.43) - open/close 29750 (time 1763156256.40 total 410.13 last 73.40) - open/close 30000 (time 1763156259.81 total 413.53 last 73.36) - open/close 30776 (time 1763156269.82 total 423.55 last 77.50) - open/close 31505 (time 1763156279.83 total 433.56 last 72.81) - open/close 32282 (time 1763156289.84 total 443.56 last 77.67) - open/close 33079 (time 1763156299.85 total 453.57 last 79.61) - open/close 33827 (time 1763156309.86 total 463.59 last 74.67) - open/close 34567 (time 1763156319.88 total 473.60 last 73.91) - open/close 35305 (time 1763156329.88 total 483.61 last 73.78) - open/close 36179 (time 1763156339.88 total 493.61 last 87.38) - open/close 37014 (time 1763156349.89 total 503.62 last 83.41) - open/close 37875 (time 1763156359.90 total 513.63 last 86.03) - open/close 38700 (time 1763156369.90 total 523.63 last 82.48) - open/close 39497 (time 1763156379.90 total 533.63 last 79.69) - open/close 40000 (time 1763156386.40 total 540.13 last 77.39) - open/close 40825 (time 1763156396.41 total 550.13 last 82.49) - open/close 41602 (time 1763156406.41 total 560.14 last 77.65) - open/close 42359 (time 1763156416.41 total 570.14 last 75.68) - open/close 43127 (time 1763156426.42 total 580.15 last 76.73) - open/close 43955 (time 1763156436.43 total 590.16 last 82.71) - open/close 44850 (time 1763156446.44 total 600.16 last 89.49) - open/close 45748 (time 1763156456.44 total 610.16 last 89.80) - open/close 46722 (time 1763156466.45 total 620.18 last 97.28) - open/close 47609 (time 1763156476.46 total 630.19 last 88.60) - open/close 48383 (time 1763156486.47 total 640.20 last 77.34) - open/close 49108 (time 1763156496.48 total 650.21 last 72.43) - open/close 49949 (time 1763156506.49 total 660.22 last 84.02) - open/close 50761 (time 1763156516.50 total 670.22 last 81.13) - open/close 51638 (time 1763156526.51 total 680.23 last 87.61) - open/close 52465 (time 1763156536.51 total 690.24 last 82.67) - open/close 53275 (time 1763156546.52 total 700.25 last 80.94) - open/close 54055 (time 1763156556.52 total 710.25 last 77.95) - open/close 54834 (time 1763156566.53 total 720.26 last 77.82) - open/close 55681 (time 1763156576.54 total 730.27 last 84.67) - open/close 56551 (time 1763156586.54 total 740.27 last 86.97) - open/close 57340 (time 1763156596.55 total 750.28 last 78.81) - open/close 58173 (time 1763156606.57 total 760.30 last 83.17) - open/close 59087 (time 1763156616.57 total 770.30 last 91.38) - open/close 60000 (time 1763156626.43 total 780.16 last 92.58) - open/close 60859 (time 1763156636.44 total 790.16 last 85.86) - open/close 61630 (time 1763156646.44 total 800.17 last 77.07) - open/close 62460 (time 1763156656.44 total 810.17 last 82.98) - open/close 63347 (time 1763156666.45 total 820.18 last 88.67) - open/close 64313 (time 1763156676.46 total 830.18 last 96.52) total: 64768 open/close in 834.69 seconds: 77.60 ops/second - /unlink 953 (time 1763156693.25 total 10.00 last 95.26) - /unlink 2016 (time 1763156703.25 total 20.01 last 106.24) - /unlink 3114 (time 1763156713.26 total 30.02 last 109.71) - /unlink 4308 (time 1763156723.27 total 40.03 last 119.24) - /unlink 5480 (time 1763156733.29 total 50.05 last 116.96) - /unlink 6582 (time 1763156743.30 total 60.06 last 110.17) - /unlink 7718 (time 1763156753.30 total 70.06 last 113.55) - /unlink 8963 (time 1763156763.31 total 80.06 last 124.45) - /unlink 10000 (time 1763156773.22 total 89.98 last 104.59) - /unlink 11006 (time 1763156783.22 total 99.98 last 100.60) - /unlink 12109 (time 1763156793.23 total 109.98 last 110.25) - /unlink 13242 (time 1763156803.23 total 119.99 last 113.24) - /unlink 14551 (time 1763156813.23 total 129.99 last 130.89) - /unlink 15796 (time 1763156823.24 total 139.99 last 124.44) - /unlink 17160 (time 1763156833.24 total 150.00 last 136.35) - /unlink 18609 (time 1763156843.24 total 160.00 last 144.86) - /unlink 19939 (time 1763156853.25 total 170.00 last 132.96) - /unlink 21145 (time 1763156863.25 total 180.01 last 120.54) - /unlink 22277 (time 1763156873.25 total 190.01 last 113.17) - /unlink 23466 (time 1763156883.26 total 200.01 last 118.87) - /unlink 24830 (time 1763156893.26 total 210.02 last 136.31) - /unlink 26303 (time 1763156903.27 total 220.02 last 147.24) - /unlink 28010 (time 1763156913.27 total 230.03 last 170.61) - /unlink 29643 (time 1763156923.27 total 240.03 last 163.28) - /unlink 30000 (time 1763156926.09 total 242.85 last 126.58) - /unlink 31293 (time 1763156936.10 total 252.86 last 129.24) - /unlink 32620 (time 1763156946.10 total 262.86 last 132.68) - /unlink 34272 (time 1763156956.10 total 272.86 last 165.17) - /unlink 36190 (time 1763156966.10 total 282.86 last 191.76) - /unlink 37967 (time 1763156976.11 total 292.86 last 177.66) - /unlink 39821 (time 1763156986.11 total 302.87 last 185.33) - /unlink 42703 (time 1763156996.11 total 312.87 last 288.15) - /unlink 45497 (time 1763157006.11 total 322.87 last 279.37) - /unlink 49004 (time 1763157016.11 total 332.87 last 350.69) - /unlink 50000 (time 1763157019.01 total 335.77 last 343.33) - /unlink 53528 (time 1763157029.02 total 345.77 last 352.67) - /unlink 56987 (time 1763157039.02 total 355.78 last 345.81) - /unlink 60000 (time 1763157047.89 total 364.65 last 339.51) - /unlink 63749 (time 1763157057.90 total 374.65 last 374.84) total: 64768 /unlink in 378.23 seconds: 171.24 ops/second Waiting for MDT destroys to complete oleg149-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 - open/close 1414 (time 1763157083.65 total 10.01 last 141.22) - open/close 3019 (time 1763157093.66 total 20.02 last 160.42) - open/close 4568 (time 1763157103.66 total 30.02 last 154.85) - open/close 6290 (time 1763157113.67 total 40.03 last 172.10) - open/close 7748 (time 1763157123.67 total 50.03 last 145.77) - open/close 9317 (time 1763157133.67 total 60.03 last 156.90) - open/close 10000 (time 1763157138.34 total 64.70 last 146.24) - open/close 11739 (time 1763157148.34 total 74.70 last 173.86) - open/close 13296 (time 1763157158.34 total 84.70 last 155.69) - open/close 15248 (time 1763157168.35 total 94.71 last 195.09) - open/close 17208 (time 1763157178.35 total 104.71 last 195.93) - open/close 18686 (time 1763157188.36 total 114.71 last 147.75) - open/close 19962 (time 1763157198.36 total 124.72 last 127.49) - open/close 21420 (time 1763157208.37 total 134.73 last 145.68) - open/close 23334 (time 1763157218.38 total 144.73 last 191.33) - open/close 25392 (time 1763157228.38 total 154.74 last 205.71) - open/close 27121 (time 1763157238.39 total 164.74 last 172.79) - open/close 28852 (time 1763157248.39 total 174.75 last 173.01) - open/close 30000 (time 1763157255.03 total 181.38 last 173.08) - open/close 31670 (time 1763157265.03 total 191.38 last 166.99) - open/close 33328 (time 1763157275.03 total 201.38 last 165.77) - open/close 34878 (time 1763157285.03 total 211.39 last 154.97) - open/close 36529 (time 1763157295.03 total 221.39 last 165.05) - open/close 38050 (time 1763157305.04 total 231.39 last 152.06) - open/close 40000 (time 1763157314.24 total 240.60 last 211.87) - open/close 41804 (time 1763157324.24 total 250.60 last 180.29) - open/close 43288 (time 1763157334.25 total 260.60 last 148.36) - open/close 45169 (time 1763157344.25 total 270.61 last 188.07) - open/close 46998 (time 1763157354.25 total 280.61 last 182.82) - open/close 48875 (time 1763157364.25 total 290.61 last 187.69) - open/close 50000 (time 1763157369.86 total 296.21 last 200.79) - open/close 51863 (time 1763157379.86 total 306.21 last 186.30) - open/close 54397 (time 1763157389.86 total 316.22 last 253.36) - open/close 56352 (time 1763157399.86 total 326.22 last 195.46) - open/close 58511 (time 1763157409.86 total 336.22 last 215.83) - open/close 60000 (time 1763157415.98 total 342.34 last 243.50) - open/close 61797 (time 1763157425.98 total 352.34 last 179.65) - open/close 63308 (time 1763157435.98 total 362.34 last 151.05) total: 64768 open/close in 372.03 seconds: 174.09 ops/second - /unlink 3446 (time 1763157457.41 total 10.00 last 344.55) - /unlink 7155 (time 1763157467.41 total 20.00 last 370.79) - /unlink 9953 (time 1763157477.41 total 30.01 last 279.78) - /unlink 12918 (time 1763157487.41 total 40.01 last 296.43) - /unlink 16866 (time 1763157497.41 total 50.01 last 394.75) - /unlink 20000 (time 1763157505.86 total 58.45 last 371.21) - /unlink 24082 (time 1763157515.86 total 68.45 last 408.16) - /unlink 27882 (time 1763157525.86 total 78.45 last 379.91) - /unlink 30000 (time 1763157530.95 total 83.55 last 415.96) - /unlink 33484 (time 1763157540.95 total 93.55 last 348.39) - /unlink 36301 (time 1763157550.95 total 103.55 last 281.69) - /unlink 39389 (time 1763157560.95 total 113.55 last 308.78) - /unlink 42712 (time 1763157570.96 total 123.55 last 332.16) - /unlink 45682 (time 1763157580.96 total 133.55 last 296.95) - /unlink 49507 (time 1763157590.96 total 143.55 last 382.48) - /unlink 54115 (time 1763157600.96 total 153.56 last 460.74) - /unlink 58679 (time 1763157610.96 total 163.56 last 456.37) - /unlink 60000 (time 1763157613.87 total 166.46 last 454.50) - /unlink 63564 (time 1763157623.87 total 176.47 last 356.27) total: 64768 /unlink in 180.66 seconds: 358.51 ops/second Waiting for MDT destroys to complete oleg149-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 - open/close 2454 (time 1763157649.80 total 10.00 last 245.32) - open/close 4658 (time 1763157659.80 total 20.01 last 220.35) - open/close 6410 (time 1763157669.81 total 30.01 last 175.15) - open/close 8682 (time 1763157679.81 total 40.01 last 227.16) - open/close 10000 (time 1763157686.19 total 46.39 last 206.56) - open/close 12407 (time 1763157696.20 total 56.40 last 240.54) - open/close 14820 (time 1763157706.20 total 66.40 last 241.14) - open/close 17886 (time 1763157716.20 total 76.40 last 306.57) - open/close 20000 (time 1763157724.03 total 84.23 last 270.06) - open/close 23397 (time 1763157734.03 total 94.24 last 339.61) - open/close 26456 (time 1763157744.04 total 104.24 last 305.80) - open/close 29101 (time 1763157754.04 total 114.24 last 264.42) - open/close 30000 (time 1763157758.16 total 118.37 last 217.98) - open/close 32305 (time 1763157768.17 total 128.37 last 230.36) - open/close 35503 (time 1763157778.17 total 138.37 last 319.74) - open/close 38722 (time 1763157788.17 total 148.37 last 321.89) - open/close 40000 (time 1763157791.51 total 151.71 last 382.52) - open/close 43431 (time 1763157801.52 total 161.72 last 343.01) - open/close 46710 (time 1763157811.52 total 171.72 last 327.88) - open/close 49810 (time 1763157821.52 total 181.72 last 309.93) - open/close 53440 (time 1763157831.52 total 191.72 last 362.97) - open/close 57029 (time 1763157841.52 total 201.72 last 358.89) - open/close 60000 (time 1763157850.76 total 210.96 last 321.55) - open/close 63393 (time 1763157860.76 total 220.96 last 339.22) total: 64768 open/close in 224.50 seconds: 288.50 ops/second - /unlink 8112 (time 1763157875.88 total 10.00 last 811.13) - /unlink 10000 (time 1763157878.08 total 12.20 last 857.67) - /unlink 17367 (time 1763157888.08 total 22.20 last 736.67) - /unlink 20000 (time 1763157891.25 total 25.37 last 830.32) - /unlink 29184 (time 1763157901.25 total 35.37 last 918.31) - /unlink 37002 (time 1763157911.25 total 45.38 last 781.73) - /unlink 40000 (time 1763157914.90 total 49.02 last 822.06) - /unlink 48368 (time 1763157924.90 total 59.02 last 836.77) - /unlink 50000 (time 1763157927.00 total 61.13 last 775.89) - /unlink 58002 (time 1763157937.00 total 71.13 last 800.17) - /unlink 60000 (time 1763157939.17 total 73.30 last 920.75) total: 64768 /unlink in 79.03 seconds: 819.51 ops/second Waiting for MDT destroys to complete oleg149-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 - open/close 4465 (time 1763158039.67 total 10.00 last 446.48) - open/close 9105 (time 1763158049.67 total 20.00 last 463.90) - open/close 10000 (time 1763158052.14 total 22.47 last 362.11) - open/close 14241 (time 1763158062.15 total 32.48 last 424.02) - open/close 18690 (time 1763158072.15 total 42.48 last 444.90) - open/close 20000 (time 1763158074.82 total 45.15 last 490.29) - open/close 25009 (time 1763158084.82 total 55.15 last 500.90) - open/close 30000 (time 1763158094.74 total 65.06 last 503.31) - open/close 34361 (time 1763158104.74 total 75.06 last 436.10) - open/close 39033 (time 1763158114.74 total 85.07 last 467.14) - open/close 43788 (time 1763158124.74 total 95.07 last 475.42) - open/close 47829 (time 1763158134.74 total 105.07 last 404.10) - open/close 50000 (time 1763158140.72 total 111.05 last 362.87) - open/close 54491 (time 1763158150.72 total 121.05 last 449.02) - open/close 58971 (time 1763158160.72 total 131.05 last 447.97) - open/close 60000 (time 1763158162.84 total 133.17 last 485.43) total: 64768 open/close in 143.05 seconds: 452.77 ops/second - /unlink 7891 (time 1763158184.07 total 10.00 last 789.06) - /unlink 10000 (time 1763158186.62 total 12.56 last 825.56) - /unlink 18102 (time 1763158196.62 total 22.56 last 810.14) - /unlink 20000 (time 1763158198.68 total 24.61 last 922.63) - /unlink 28880 (time 1763158208.68 total 34.61 last 887.97) - /unlink 37536 (time 1763158218.68 total 44.61 last 865.52) - /unlink 40000 (time 1763158222.46 total 48.40 last 651.55) - /unlink 49165 (time 1763158232.46 total 58.40 last 916.41) - /unlink 58117 (time 1763158242.46 total 68.40 last 895.17) - /unlink 60000 (time 1763158244.93 total 70.86 last 763.77) total: 64768 /unlink in 77.27 seconds: 838.22 ops/second Waiting for MDT destroys to complete oleg149-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg149-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg149-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg149-server stop mds service on oleg149-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg149-server stop mds service on oleg149-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg149-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg149-server' oleg149-server: oleg149-server.virtnet: executing unload_modules_local oleg149-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg149-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg149-client: oleg149-client: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-server: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-client: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-server: ssh exited with exit code 2 PASS 106 (4376s) Stopping clients: oleg149-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg149-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg149-client: oleg149-server: ssh exited with exit code 2 oleg149-server: oleg149-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg149-server' oleg149-server: oleg149-server.virtnet: executing load_modules_local oleg149-server: Loading modules from /home/green/git/lustre-release/lustre oleg149-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg149-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg149-server: detected 4 online CPUs by sysfs oleg149-server: MODOPTS_LIBCFS= oleg149-server: Force libcfs to create 2 CPU partitions oleg149-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg149-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4568 sec ========== 17:12:36 (1763158356) === conf-sanity: start cleanup 17:12:37 (1763158357) === === conf-sanity: finish cleanup 17:12:38 (1763158358) ===