-----============= acceptance-small: conf-sanity ============----- Fri Oct 10 15:46:20 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg317-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg317-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server oleg317-server: oleg317-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg317-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg317-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:49:01 (1760125741) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) oleg317-server: oleg317-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg317-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg317-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1030 (time 1760125828.17 total 10.00 last 102.95) - open/close 2091 (time 1760125838.18 total 20.01 last 105.99) - open/close 3211 (time 1760125848.19 total 30.03 last 111.84) - open/close 4305 (time 1760125858.20 total 40.03 last 109.38) - open/close 5416 (time 1760125868.20 total 50.04 last 111.02) - open/close 6659 (time 1760125878.21 total 60.04 last 124.22) - open/close 7737 (time 1760125888.21 total 70.05 last 107.76) - open/close 8919 (time 1760125898.22 total 80.05 last 118.14) - open/close 10000 (time 1760125907.65 total 89.48 last 114.63) - open/close 11070 (time 1760125917.65 total 99.48 last 106.99) - open/close 12137 (time 1760125927.66 total 109.49 last 106.61) - open/close 13297 (time 1760125937.66 total 119.50 last 115.96) - open/close 14347 (time 1760125947.68 total 129.51 last 104.81) - open/close 15349 (time 1760125957.69 total 139.52 last 100.10) - open/close 16380 (time 1760125967.69 total 149.53 last 103.07) - open/close 17502 (time 1760125977.69 total 159.53 last 112.19) - open/close 18560 (time 1760125987.70 total 169.53 last 105.73) - open/close 19628 (time 1760125997.70 total 179.54 last 106.78) - open/close 20000 (time 1760126001.25 total 183.08 last 104.86) - open/close 21162 (time 1760126011.25 total 193.08 last 116.19) - open/close 22199 (time 1760126021.25 total 203.09 last 103.67) - open/close 23212 (time 1760126031.26 total 213.09 last 101.27) - open/close 24252 (time 1760126041.26 total 223.09 last 103.96) - open/close 25268 (time 1760126051.27 total 233.10 last 101.53) - open/close 26288 (time 1760126061.27 total 243.10 last 101.99) - open/close 27248 (time 1760126071.27 total 253.10 last 95.97) - open/close 28278 (time 1760126081.27 total 263.11 last 102.98) - open/close 29269 (time 1760126091.28 total 273.11 last 99.05) - open/close 30000 (time 1760126098.15 total 279.98 last 106.37) - open/close 31084 (time 1760126108.16 total 289.99 last 108.31) - open/close 32170 (time 1760126118.16 total 300.00 last 108.52) - open/close 33237 (time 1760126128.17 total 310.00 last 106.66) - open/close 34273 (time 1760126138.18 total 320.01 last 103.52) - open/close 35307 (time 1760126148.18 total 330.02 last 103.33) - open/close 36361 (time 1760126158.19 total 340.03 last 105.30) - open/close 37462 (time 1760126168.20 total 350.03 last 110.01) - open/close 38583 (time 1760126178.22 total 360.05 last 111.92) - open/close 39761 (time 1760126188.22 total 370.06 last 117.73) - open/close 40000 (time 1760126190.30 total 372.13 last 115.05) - open/close 41136 (time 1760126200.30 total 382.14 last 113.54) - open/close 42257 (time 1760126210.30 total 392.14 last 112.09) - open/close 43298 (time 1760126220.32 total 402.15 last 103.99) - open/close 44469 (time 1760126230.32 total 412.15 last 117.05) - open/close 45535 (time 1760126240.32 total 422.15 last 106.59) - open/close 46548 (time 1760126250.32 total 432.16 last 101.25) - open/close 47618 (time 1760126260.33 total 442.16 last 106.96) - open/close 48802 (time 1760126270.33 total 452.17 last 118.32) - open/close 49931 (time 1760126280.34 total 462.18 last 112.82) - open/close 51152 (time 1760126290.34 total 472.18 last 122.09) - open/close 52571 (time 1760126300.35 total 482.18 last 141.83) - open/close 53924 (time 1760126310.35 total 492.19 last 135.23) - open/close 55295 (time 1760126320.36 total 502.19 last 137.05) - open/close 56697 (time 1760126330.36 total 512.20 last 140.14) - open/close 58234 (time 1760126340.37 total 522.20 last 153.59) - open/close 59484 (time 1760126350.37 total 532.20 last 124.98) - open/close 60000 (time 1760126353.82 total 535.66 last 149.52) - open/close 61393 (time 1760126363.83 total 545.66 last 139.21) - open/close 62740 (time 1760126373.84 total 555.67 last 134.55) - open/close 64022 (time 1760126383.85 total 565.68 last 128.13) total: 64768 open/close in 571.57 seconds: 113.32 ops/second - /unlink 2212 (time 1760126401.70 total 10.00 last 221.13) - /unlink 4686 (time 1760126411.70 total 20.00 last 247.37) - /unlink 6251 (time 1760126421.70 total 30.01 last 156.49) - /unlink 7671 (time 1760126431.70 total 40.01 last 141.98) - /unlink 9160 (time 1760126441.71 total 50.01 last 148.81) - /unlink 10000 (time 1760126447.30 total 55.60 last 150.30) - /unlink 11570 (time 1760126457.30 total 65.61 last 156.87) - /unlink 13138 (time 1760126467.31 total 75.61 last 156.78) - /unlink 14867 (time 1760126477.31 total 85.61 last 172.87) - /unlink 16534 (time 1760126487.32 total 95.62 last 166.57) - /unlink 18263 (time 1760126497.32 total 105.62 last 172.89) - /unlink 20000 (time 1760126507.04 total 115.35 last 178.58) - /unlink 21679 (time 1760126517.05 total 125.35 last 167.84) - /unlink 23523 (time 1760126527.05 total 135.35 last 184.37) - /unlink 25242 (time 1760126537.05 total 145.35 last 171.86) - /unlink 26904 (time 1760126547.05 total 155.36 last 166.14) - /unlink 28639 (time 1760126557.06 total 165.36 last 173.41) - /unlink 30000 (time 1760126565.16 total 173.47 last 167.95) - /unlink 31784 (time 1760126575.17 total 183.47 last 178.30) - /unlink 33522 (time 1760126585.17 total 193.47 last 173.77) - /unlink 35158 (time 1760126595.17 total 203.48 last 163.52) - /unlink 36842 (time 1760126605.18 total 213.48 last 168.36) - /unlink 38457 (time 1760126615.18 total 223.48 last 161.48) - /unlink 40000 (time 1760126624.32 total 232.62 last 168.88) - /unlink 41735 (time 1760126634.32 total 242.62 last 173.46) - /unlink 43516 (time 1760126644.32 total 252.62 last 178.07) - /unlink 45428 (time 1760126654.32 total 262.63 last 191.10) - /unlink 47068 (time 1760126664.32 total 272.63 last 164.00) - /unlink 48947 (time 1760126674.33 total 282.63 last 187.83) - /unlink 50000 (time 1760126679.84 total 288.15 last 190.93) - /unlink 51742 (time 1760126689.84 total 298.15 last 174.18) - /unlink 53508 (time 1760126699.85 total 308.15 last 176.50) - /unlink 54823 (time 1760126709.93 total 318.23 last 130.49) - /unlink 56069 (time 1760126719.93 total 328.24 last 124.53) - /unlink 57646 (time 1760126729.94 total 338.24 last 157.67) - /unlink 59178 (time 1760126739.98 total 348.28 last 152.51) - /unlink 60000 (time 1760126746.84 total 355.14 last 119.89) - /unlink 61312 (time 1760126756.88 total 365.19 last 130.59) - /unlink 62978 (time 1760126766.88 total 375.19 last 166.59) - /unlink 64705 (time 1760126776.89 total 385.19 last 172.64) total: 64768 /unlink in 385.57 seconds: 167.98 ops/second Waiting for MDT destroys to complete oleg317-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 - open/close 957 (time 1760126801.63 total 10.01 last 95.64) - open/close 1928 (time 1760126811.63 total 20.01 last 97.08) - open/close 2867 (time 1760126821.64 total 30.01 last 93.86) - open/close 3884 (time 1760126831.64 total 40.02 last 101.65) - open/close 4831 (time 1760126841.65 total 50.02 last 94.65) - open/close 5917 (time 1760126851.65 total 60.02 last 108.58) - open/close 6962 (time 1760126861.65 total 70.03 last 104.47) - open/close 8016 (time 1760126871.66 total 80.04 last 105.30) - open/close 8946 (time 1760126881.67 total 90.04 last 92.94) - open/close 9913 (time 1760126891.67 total 100.05 last 96.67) - open/close 10887 (time 1760126901.68 total 110.06 last 97.31) - open/close 11881 (time 1760126911.69 total 120.06 last 99.34) - open/close 12884 (time 1760126921.69 total 130.07 last 100.26) - open/close 13892 (time 1760126931.70 total 140.07 last 100.77) - open/close 14878 (time 1760126941.70 total 150.07 last 98.56) - open/close 15846 (time 1760126951.70 total 160.08 last 96.78) - open/close 16808 (time 1760126961.70 total 170.08 last 96.17) - open/close 17735 (time 1760126971.72 total 180.09 last 92.57) - open/close 18625 (time 1760126981.72 total 190.10 last 88.97) - open/close 19437 (time 1760126991.73 total 200.10 last 81.16) - open/close 20000 (time 1760126997.48 total 205.85 last 97.91) - open/close 20978 (time 1760127007.49 total 215.87 last 97.65) - open/close 21971 (time 1760127017.50 total 225.87 last 99.23) - open/close 22885 (time 1760127027.50 total 235.88 last 91.37) - open/close 23789 (time 1760127037.51 total 245.88 last 90.38) - open/close 24788 (time 1760127047.51 total 255.88 last 99.85) - open/close 25759 (time 1760127057.53 total 265.90 last 96.95) - open/close 26822 (time 1760127067.53 total 275.91 last 106.22) - open/close 27758 (time 1760127077.54 total 285.91 last 93.54) - open/close 28703 (time 1760127087.54 total 295.92 last 94.48) - open/close 29453 (time 1760127097.55 total 305.93 last 74.92) - open/close 30000 (time 1760127106.07 total 314.45 last 64.21) - open/close 30961 (time 1760127116.08 total 324.45 last 96.06) - open/close 31852 (time 1760127126.08 total 334.46 last 89.04) - open/close 32823 (time 1760127136.09 total 344.46 last 97.07) - open/close 33762 (time 1760127146.10 total 354.47 last 93.77) - open/close 34492 (time 1760127156.10 total 364.48 last 73.00) - open/close 35404 (time 1760127166.10 total 374.48 last 91.18) - open/close 36302 (time 1760127176.11 total 384.48 last 89.77) - open/close 37233 (time 1760127186.11 total 394.49 last 93.06) - open/close 38073 (time 1760127196.12 total 404.49 last 83.97) - open/close 39017 (time 1760127206.12 total 414.49 last 94.36) - open/close 40000 (time 1760127215.83 total 424.20 last 101.21) - open/close 41042 (time 1760127225.84 total 434.21 last 104.15) - open/close 42058 (time 1760127235.84 total 444.21 last 101.59) - open/close 43087 (time 1760127245.85 total 454.22 last 102.79) - open/close 43873 (time 1760127255.85 total 464.22 last 78.60) - open/close 44671 (time 1760127265.85 total 474.23 last 79.76) - open/close 45636 (time 1760127275.86 total 484.23 last 96.43) - open/close 46531 (time 1760127285.86 total 494.23 last 89.50) - open/close 47448 (time 1760127295.87 total 504.25 last 91.58) - open/close 48412 (time 1760127305.88 total 514.26 last 96.32) - open/close 49363 (time 1760127315.89 total 524.26 last 95.03) - open/close 50000 (time 1760127323.04 total 531.42 last 89.07) - open/close 50918 (time 1760127333.04 total 541.42 last 91.78) - open/close 51826 (time 1760127343.05 total 551.42 last 90.76) - open/close 52771 (time 1760127353.05 total 561.43 last 94.47) - open/close 53702 (time 1760127363.06 total 571.43 last 93.05) - open/close 54622 (time 1760127373.06 total 581.44 last 91.95) - open/close 55512 (time 1760127383.07 total 591.44 last 88.93) - open/close 56429 (time 1760127393.07 total 601.45 last 91.67) - open/close 57406 (time 1760127403.08 total 611.45 last 97.67) - open/close 58266 (time 1760127413.08 total 621.45 last 85.99) - open/close 59143 (time 1760127423.08 total 631.46 last 87.65) - open/close 60000 (time 1760127432.33 total 640.70 last 92.68) - open/close 60836 (time 1760127442.34 total 650.71 last 83.53) - open/close 61712 (time 1760127452.35 total 660.72 last 87.50) - open/close 62509 (time 1760127462.36 total 670.74 last 79.61) - open/close 63385 (time 1760127472.37 total 680.74 last 87.55) - open/close 64282 (time 1760127482.37 total 690.74 last 89.68) total: 64768 open/close in 696.21 seconds: 93.03 ops/second - /unlink 920 (time 1760127501.03 total 10.01 last 91.92) - /unlink 1845 (time 1760127511.03 total 20.01 last 92.46) - /unlink 2744 (time 1760127521.03 total 30.02 last 89.87) - /unlink 3695 (time 1760127531.04 total 40.02 last 95.04) - /unlink 4639 (time 1760127541.04 total 50.03 last 94.35) - /unlink 5576 (time 1760127551.06 total 60.04 last 93.57) - /unlink 6495 (time 1760127561.06 total 70.04 last 91.89) - /unlink 7359 (time 1760127571.07 total 80.05 last 86.34) - /unlink 8227 (time 1760127581.07 total 90.05 last 86.80) - /unlink 9129 (time 1760127591.07 total 100.06 last 90.16) - /unlink 9987 (time 1760127601.07 total 110.06 last 85.78) - /unlink 10927 (time 1760127611.08 total 120.07 last 93.90) - /unlink 11878 (time 1760127621.09 total 130.07 last 95.05) - /unlink 12711 (time 1760127631.09 total 140.08 last 83.27) - /unlink 13623 (time 1760127641.09 total 150.08 last 91.19) - /unlink 14559 (time 1760127651.11 total 160.09 last 93.50) - /unlink 15504 (time 1760127661.11 total 170.09 last 94.49) - /unlink 16484 (time 1760127671.12 total 180.10 last 97.89) - /unlink 17476 (time 1760127681.12 total 190.10 last 99.16) - /unlink 18357 (time 1760127691.12 total 200.11 last 88.06) - /unlink 19432 (time 1760127701.13 total 210.11 last 107.49) - /unlink 20000 (time 1760127707.27 total 216.25 last 92.52) - /unlink 20896 (time 1760127717.27 total 226.26 last 89.54) - /unlink 21833 (time 1760127727.28 total 236.26 last 93.67) - /unlink 22815 (time 1760127737.28 total 246.27 last 98.11) - /unlink 23735 (time 1760127747.29 total 256.28 last 91.94) - /unlink 24719 (time 1760127757.29 total 266.28 last 98.38) - /unlink 25700 (time 1760127767.30 total 276.28 last 98.08) - /unlink 26763 (time 1760127777.30 total 286.29 last 106.21) - /unlink 28025 (time 1760127787.31 total 296.30 last 126.10) - /unlink 29188 (time 1760127797.32 total 306.30 last 116.27) - /unlink 30000 (time 1760127803.67 total 312.65 last 127.82) - /unlink 31049 (time 1760127813.67 total 322.66 last 104.85) - /unlink 31986 (time 1760127823.68 total 332.66 last 93.65) - /unlink 32972 (time 1760127833.68 total 342.66 last 98.57) - /unlink 33947 (time 1760127843.69 total 352.67 last 97.41) - /unlink 34876 (time 1760127853.69 total 362.68 last 92.88) - /unlink 35748 (time 1760127863.70 total 372.68 last 87.15) - /unlink 36657 (time 1760127873.70 total 382.69 last 90.85) - /unlink 37537 (time 1760127883.72 total 392.70 last 87.89) - /unlink 38489 (time 1760127893.72 total 402.70 last 95.20) - /unlink 39464 (time 1760127903.72 total 412.71 last 97.42) - /unlink 40000 (time 1760127909.26 total 418.24 last 96.83) - /unlink 40976 (time 1760127919.27 total 428.25 last 97.54) - /unlink 41936 (time 1760127929.27 total 438.25 last 95.98) - /unlink 42922 (time 1760127939.27 total 448.26 last 98.56) - /unlink 43913 (time 1760127949.29 total 458.28 last 98.88) - /unlink 44765 (time 1760127959.30 total 468.28 last 85.16) - /unlink 45715 (time 1760127969.30 total 478.29 last 94.95) - /unlink 46647 (time 1760127979.32 total 488.30 last 93.09) - /unlink 47593 (time 1760127989.34 total 498.32 last 94.38) - /unlink 48445 (time 1760127999.35 total 508.34 last 85.10) - /unlink 49343 (time 1760128009.36 total 518.34 last 89.77) - /unlink 50000 (time 1760128015.86 total 524.84 last 101.08) - /unlink 50875 (time 1760128025.87 total 534.85 last 87.41) - /unlink 51884 (time 1760128035.87 total 544.85 last 100.86) - /unlink 52846 (time 1760128045.87 total 554.85 last 96.19) - /unlink 53725 (time 1760128055.88 total 564.86 last 87.83) - /unlink 54581 (time 1760128065.89 total 574.87 last 85.53) - /unlink 55548 (time 1760128075.89 total 584.87 last 96.70) - /unlink 56540 (time 1760128085.89 total 594.87 last 99.20) - /unlink 57591 (time 1760128095.89 total 604.87 last 105.07) - /unlink 58481 (time 1760128105.89 total 614.88 last 88.98) - /unlink 59428 (time 1760128115.90 total 624.88 last 94.64) - /unlink 60000 (time 1760128122.12 total 631.10 last 91.98) - /unlink 60977 (time 1760128132.12 total 641.11 last 97.65) - /unlink 61900 (time 1760128142.13 total 651.11 last 92.23) - /unlink 62852 (time 1760128152.14 total 661.12 last 95.13) - /unlink 63851 (time 1760128162.14 total 671.13 last 99.86) total: 64768 /unlink in 680.44 seconds: 95.19 ops/second Waiting for MDT destroys to complete oleg317-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 - open/close 952 (time 1760128196.34 total 10.01 last 95.14) - open/close 1902 (time 1760128206.35 total 20.01 last 94.93) - open/close 2845 (time 1760128216.35 total 30.02 last 94.23) - open/close 3887 (time 1760128226.36 total 40.03 last 104.13) - open/close 4918 (time 1760128236.37 total 50.03 last 103.04) - open/close 5944 (time 1760128246.37 total 60.04 last 102.56) - open/close 6941 (time 1760128256.38 total 70.05 last 99.62) - open/close 7969 (time 1760128266.38 total 80.05 last 102.74) - open/close 8917 (time 1760128276.39 total 90.05 last 94.78) - open/close 9958 (time 1760128286.39 total 100.06 last 104.05) - open/close 10940 (time 1760128296.40 total 110.06 last 98.15) - open/close 11927 (time 1760128306.40 total 120.07 last 98.66) - open/close 13012 (time 1760128316.40 total 130.07 last 108.46) - open/close 14185 (time 1760128326.41 total 140.08 last 117.26) - open/close 15286 (time 1760128336.42 total 150.09 last 109.91) - open/close 16348 (time 1760128346.43 total 160.10 last 106.09) - open/close 17378 (time 1760128356.44 total 170.11 last 102.93) - open/close 18422 (time 1760128366.45 total 180.12 last 104.32) - open/close 19451 (time 1760128376.45 total 190.12 last 102.85) - open/close 20000 (time 1760128381.21 total 194.88 last 115.33) - open/close 21105 (time 1760128391.22 total 204.89 last 110.44) - open/close 22130 (time 1760128401.23 total 214.89 last 102.44) - open/close 23159 (time 1760128411.23 total 224.90 last 102.83) - open/close 24225 (time 1760128421.23 total 234.90 last 106.57) - open/close 25332 (time 1760128431.24 total 244.91 last 110.62) - open/close 26354 (time 1760128441.24 total 254.91 last 102.19) - open/close 27546 (time 1760128451.25 total 264.91 last 119.15) - open/close 28851 (time 1760128461.25 total 274.92 last 130.49) - open/close 30000 (time 1760128470.96 total 284.62 last 118.36) - open/close 31036 (time 1760128480.96 total 294.63 last 103.53) - open/close 32149 (time 1760128490.97 total 304.64 last 111.24) - open/close 33293 (time 1760128500.97 total 314.64 last 114.37) - open/close 34366 (time 1760128510.97 total 324.64 last 107.28) - open/close 35492 (time 1760128520.98 total 334.64 last 112.54) - open/close 36743 (time 1760128530.98 total 344.65 last 125.01) - open/close 37976 (time 1760128540.99 total 354.65 last 123.27) - open/close 39162 (time 1760128551.00 total 364.66 last 118.48) - open/close 40000 (time 1760128556.87 total 370.54 last 142.61) - open/close 41439 (time 1760128566.88 total 380.54 last 143.86) - open/close 42569 (time 1760128576.88 total 390.54 last 112.99) - open/close 43690 (time 1760128586.88 total 400.55 last 112.06) - open/close 44885 (time 1760128596.88 total 410.55 last 119.49) - open/close 46251 (time 1760128606.88 total 420.55 last 136.57) - open/close 47347 (time 1760128616.89 total 430.55 last 109.57) - open/close 48549 (time 1760128626.89 total 440.56 last 120.15) - open/close 49795 (time 1760128636.89 total 450.56 last 124.55) - open/close 50921 (time 1760128646.90 total 460.57 last 112.54) - open/close 51929 (time 1760128656.91 total 470.57 last 100.75) - open/close 52981 (time 1760128666.91 total 480.57 last 105.18) - open/close 54057 (time 1760128676.91 total 490.58 last 107.55) - open/close 55047 (time 1760128686.92 total 500.58 last 98.95) - open/close 55996 (time 1760128696.92 total 510.58 last 94.89) - open/close 56986 (time 1760128706.92 total 520.59 last 98.98) - open/close 58010 (time 1760128716.93 total 530.60 last 102.30) - open/close 58980 (time 1760128726.94 total 540.61 last 96.89) - open/close 60000 (time 1760128736.85 total 550.52 last 102.95) - open/close 61085 (time 1760128746.85 total 560.52 last 108.47) - open/close 62040 (time 1760128756.86 total 570.52 last 95.45) - open/close 62983 (time 1760128766.86 total 580.53 last 94.26) - open/close 63912 (time 1760128776.86 total 590.53 last 92.89) total: 64768 open/close in 599.71 seconds: 108.00 ops/second - /unlink 920 (time 1760128799.21 total 10.01 last 91.88) - /unlink 1792 (time 1760128809.21 total 20.01 last 87.19) - /unlink 2826 (time 1760128819.22 total 30.02 last 103.35) - /unlink 3893 (time 1760128829.23 total 40.03 last 106.61) - /unlink 4845 (time 1760128839.23 total 50.03 last 95.15) - /unlink 5739 (time 1760128849.24 total 60.04 last 89.34) - /unlink 6685 (time 1760128859.25 total 70.05 last 94.54) - /unlink 7668 (time 1760128869.25 total 80.05 last 98.30) - /unlink 8625 (time 1760128879.25 total 90.05 last 95.63) - /unlink 9688 (time 1760128889.26 total 100.06 last 106.25) - /unlink 10000 (time 1760128892.32 total 103.12 last 101.94) - /unlink 11122 (time 1760128902.33 total 113.13 last 112.07) - /unlink 12192 (time 1760128912.33 total 123.13 last 106.99) - /unlink 13641 (time 1760128922.33 total 133.13 last 144.89) - /unlink 15951 (time 1760128932.33 total 143.13 last 230.96) - /unlink 18288 (time 1760128942.34 total 153.14 last 233.63) - /unlink 20000 (time 1760128949.95 total 160.75 last 224.94) - /unlink 22356 (time 1760128959.95 total 170.75 last 235.50) - /unlink 24275 (time 1760128969.96 total 180.76 last 191.82) - /unlink 26071 (time 1760128979.98 total 190.78 last 179.21) - /unlink 27953 (time 1760128989.98 total 200.78 last 188.18) - /unlink 29904 (time 1760128999.99 total 210.79 last 194.97) - /unlink 31889 (time 1760129009.99 total 220.79 last 198.47) - /unlink 33837 (time 1760129019.99 total 230.79 last 194.72) - /unlink 35918 (time 1760129029.99 total 240.79 last 208.09) - /unlink 37910 (time 1760129040.00 total 250.80 last 199.07) - /unlink 40000 (time 1760129049.57 total 260.37 last 218.36) - /unlink 42272 (time 1760129059.57 total 270.37 last 227.14) - /unlink 44392 (time 1760129069.57 total 280.37 last 211.98) - /unlink 46704 (time 1760129079.57 total 290.38 last 231.15) - /unlink 48983 (time 1760129089.57 total 300.38 last 227.90) - /unlink 50000 (time 1760129093.58 total 304.38 last 253.94) - /unlink 52959 (time 1760129103.58 total 314.38 last 295.85) - /unlink 55377 (time 1760129113.58 total 324.39 last 241.71) - /unlink 57790 (time 1760129123.59 total 334.39 last 241.25) - /unlink 60000 (time 1760129133.44 total 344.24 last 224.28) - /unlink 62652 (time 1760129143.45 total 354.25 last 265.03) total: 64768 /unlink in 363.11 seconds: 178.37 ops/second Waiting for MDT destroys to complete oleg317-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 - open/close 1506 (time 1760129174.41 total 10.00 last 150.54) - open/close 2986 (time 1760129184.42 total 20.01 last 147.98) - open/close 4428 (time 1760129194.42 total 30.01 last 144.14) - open/close 6126 (time 1760129204.43 total 40.02 last 169.66) - open/close 8103 (time 1760129214.43 total 50.02 last 197.68) - open/close 9509 (time 1760129224.43 total 60.02 last 140.58) - open/close 10000 (time 1760129227.11 total 62.70 last 183.08) - open/close 11749 (time 1760129237.11 total 72.70 last 174.87) - open/close 13653 (time 1760129247.12 total 82.71 last 190.32) - open/close 15711 (time 1760129257.12 total 92.71 last 205.76) - open/close 17743 (time 1760129267.12 total 102.71 last 203.14) - open/close 19604 (time 1760129277.13 total 112.72 last 186.04) - open/close 20000 (time 1760129279.20 total 114.79 last 191.22) - open/close 21807 (time 1760129289.20 total 124.79 last 180.69) - open/close 23331 (time 1760129299.20 total 134.79 last 152.39) - open/close 25060 (time 1760129309.20 total 144.79 last 172.85) - open/close 27068 (time 1760129319.20 total 154.79 last 200.73) - open/close 28648 (time 1760129329.21 total 164.80 last 157.96) - open/close 30000 (time 1760129334.68 total 170.27 last 247.16) - open/close 31741 (time 1760129344.68 total 180.27 last 174.08) - open/close 33869 (time 1760129354.68 total 190.27 last 212.68) - open/close 36326 (time 1760129364.69 total 200.28 last 245.62) - open/close 38437 (time 1760129374.69 total 210.28 last 211.01) - open/close 40000 (time 1760129382.38 total 217.97 last 203.34) - open/close 42409 (time 1760129392.38 total 227.97 last 240.85) - open/close 45312 (time 1760129402.38 total 237.97 last 290.23) - open/close 48612 (time 1760129412.38 total 247.97 last 329.94) - open/close 50000 (time 1760129417.57 total 253.16 last 267.63) - open/close 53437 (time 1760129427.57 total 263.16 last 343.67) - open/close 57060 (time 1760129437.57 total 273.16 last 362.25) - open/close 60000 (time 1760129445.71 total 281.30 last 361.47) - open/close 62775 (time 1760129455.71 total 291.30 last 277.33) total: 64768 open/close in 301.20 seconds: 215.03 ops/second - /unlink 4733 (time 1760129477.32 total 10.00 last 473.27) - /unlink 9689 (time 1760129487.32 total 20.00 last 495.53) - /unlink 14538 (time 1760129497.32 total 30.00 last 484.85) - /unlink 19229 (time 1760129507.32 total 40.00 last 469.05) - /unlink 23242 (time 1760129517.33 total 50.01 last 401.23) - /unlink 27535 (time 1760129527.33 total 60.01 last 429.28) - /unlink 30000 (time 1760129533.88 total 66.56 last 376.20) - /unlink 33265 (time 1760129543.88 total 76.56 last 326.39) - /unlink 37318 (time 1760129553.88 total 86.56 last 405.25) - /unlink 40000 (time 1760129562.12 total 94.80 last 325.72) - /unlink 43079 (time 1760129572.12 total 104.80 last 307.89) - /unlink 46265 (time 1760129582.13 total 114.81 last 318.11) - /unlink 49656 (time 1760129592.13 total 124.81 last 339.10) - /unlink 53147 (time 1760129602.14 total 134.82 last 348.99) - /unlink 56933 (time 1760129612.14 total 144.82 last 378.53) - /unlink 60000 (time 1760129618.70 total 151.38 last 467.33) - /unlink 64374 (time 1760129628.71 total 161.39 last 437.22) total: 64768 /unlink in 162.31 seconds: 399.03 ops/second Waiting for MDT destroys to complete oleg317-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 - open/close 1628 (time 1760129650.88 total 10.00 last 162.73) - open/close 3170 (time 1760129660.88 total 20.01 last 154.16) - open/close 4762 (time 1760129670.88 total 30.01 last 159.18) - open/close 6409 (time 1760129680.89 total 40.01 last 164.68) - open/close 8183 (time 1760129690.89 total 50.01 last 177.40) - open/close 10000 (time 1760129700.62 total 59.74 last 186.67) - open/close 11702 (time 1760129710.62 total 69.75 last 170.17) - open/close 13584 (time 1760129720.63 total 79.75 last 188.04) - open/close 15346 (time 1760129730.63 total 89.75 last 176.19) - open/close 17204 (time 1760129740.63 total 99.76 last 185.76) - open/close 19053 (time 1760129750.64 total 109.76 last 184.82) - open/close 20000 (time 1760129755.60 total 114.73 last 190.68) - open/close 22195 (time 1760129765.61 total 124.73 last 219.42) - open/close 24207 (time 1760129775.61 total 134.73 last 201.18) - open/close 25944 (time 1760129785.61 total 144.74 last 173.62) - open/close 27937 (time 1760129795.61 total 154.74 last 199.26) - open/close 30000 (time 1760129804.66 total 163.78 last 228.09) - open/close 32282 (time 1760129814.66 total 173.79 last 228.12) - open/close 35629 (time 1760129824.66 total 183.79 last 334.62) - open/close 38376 (time 1760129834.67 total 193.79 last 274.63) - open/close 40000 (time 1760129841.24 total 200.36 last 247.15) - open/close 42526 (time 1760129851.24 total 210.37 last 252.53) - open/close 44390 (time 1760129861.24 total 220.37 last 186.38) - open/close 47167 (time 1760129871.24 total 230.37 last 277.64) - open/close 49801 (time 1760129881.25 total 240.37 last 263.36) - open/close 52939 (time 1760129891.25 total 250.37 last 313.75) - open/close 55702 (time 1760129901.25 total 260.37 last 276.25) - open/close 58537 (time 1760129911.25 total 270.38 last 283.43) - open/close 60000 (time 1760129915.59 total 274.71 last 337.23) - open/close 63067 (time 1760129925.59 total 284.72 last 306.65) total: 64768 open/close in 291.29 seconds: 222.35 ops/second - /unlink 4699 (time 1760129943.70 total 10.00 last 469.88) - /unlink 10000 (time 1760129953.51 total 19.81 last 540.55) - /unlink 15399 (time 1760129963.51 total 29.81 last 539.83) - /unlink 20000 (time 1760129972.38 total 38.67 last 518.94) - /unlink 25430 (time 1760129982.38 total 48.67 last 543.00) - /unlink 30000 (time 1760129989.41 total 55.71 last 649.44) - /unlink 35548 (time 1760129999.41 total 65.71 last 554.74) - /unlink 39718 (time 1760130009.41 total 75.71 last 416.98) - /unlink 44196 (time 1760130019.41 total 85.71 last 447.78) - /unlink 48670 (time 1760130029.42 total 95.72 last 447.32) - /unlink 50000 (time 1760130032.43 total 98.73 last 440.96) - /unlink 54168 (time 1760130042.43 total 108.73 last 416.78) - /unlink 58497 (time 1760130052.43 total 118.73 last 432.89) - /unlink 60000 (time 1760130055.83 total 122.12 last 443.12) total: 64768 /unlink in 130.03 seconds: 498.08 ops/second Waiting for MDT destroys to complete oleg317-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local oleg317-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 PASS 106 (4366s) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4554 sec ========== 17:02:16 (1760130136) === conf-sanity: start cleanup 17:02:16 (1760130136) === === conf-sanity: finish cleanup 17:02:17 (1760130137) ===