-----============= acceptance-small: conf-sanity ============----- Tue Nov 25 16:26:50 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg411-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball Stopping clients: oleg411-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg411-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg411-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg411-server oleg411-server: oleg411-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg411-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg411-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:28:35 (1764106115) Stopping clients: oleg411-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg411-client.virtnet /mnt/lustre2 (opts:-f) oleg411-server: oleg411-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg411-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg411-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1699 (time 1764106183.21 total 10.00 last 169.90) - open/close 3478 (time 1764106193.21 total 20.01 last 177.80) - open/close 4967 (time 1764106203.21 total 30.01 last 148.89) - open/close 6832 (time 1764106213.22 total 40.02 last 186.30) - open/close 8655 (time 1764106223.23 total 50.02 last 182.23) - open/close 10000 (time 1764106231.21 total 58.01 last 168.41) - open/close 11760 (time 1764106241.22 total 68.01 last 175.93) - open/close 13548 (time 1764106251.22 total 78.01 last 178.75) - open/close 16039 (time 1764106261.23 total 88.02 last 248.92) - open/close 18241 (time 1764106271.23 total 98.02 last 220.18) - open/close 20000 (time 1764106279.04 total 105.84 last 225.06) - open/close 22129 (time 1764106289.05 total 115.84 last 212.82) - open/close 24167 (time 1764106299.05 total 125.85 last 203.71) - open/close 26225 (time 1764106309.05 total 135.85 last 205.79) - open/close 28283 (time 1764106319.06 total 145.85 last 205.69) - open/close 30000 (time 1764106326.99 total 153.78 last 216.59) - open/close 32505 (time 1764106336.99 total 163.78 last 250.45) - open/close 34931 (time 1764106346.99 total 173.78 last 242.57) - open/close 36889 (time 1764106356.99 total 183.79 last 195.73) - open/close 38926 (time 1764106366.99 total 193.79 last 203.65) - open/close 40000 (time 1764106373.54 total 200.34 last 164.03) - open/close 41641 (time 1764106383.55 total 210.34 last 164.02) - open/close 43500 (time 1764106393.55 total 220.34 last 185.86) - open/close 45198 (time 1764106403.55 total 230.35 last 169.73) - open/close 47389 (time 1764106413.55 total 240.35 last 219.10) - open/close 50000 (time 1764106421.39 total 248.18 last 333.26) - open/close 52765 (time 1764106431.39 total 258.18 last 276.43) - open/close 55325 (time 1764106441.39 total 268.18 last 255.99) - open/close 57571 (time 1764106451.40 total 278.19 last 224.47) - open/close 59926 (time 1764106461.40 total 288.19 last 235.48) - open/close 63131 (time 1764106471.40 total 298.19 last 320.44) total: 64768 open/close in 303.49 seconds: 213.41 ops/second - /unlink 4938 (time 1764106488.36 total 10.00 last 493.76) - /unlink 10000 (time 1764106497.91 total 19.55 last 530.15) - /unlink 15409 (time 1764106507.91 total 29.55 last 540.90) - /unlink 20000 (time 1764106516.72 total 38.35 last 521.38) - /unlink 25449 (time 1764106526.72 total 48.36 last 544.85) - /unlink 29808 (time 1764106536.73 total 58.36 last 435.61) - /unlink 34677 (time 1764106546.73 total 68.37 last 486.72) - /unlink 40000 (time 1764106555.92 total 77.55 last 579.35) - /unlink 48440 (time 1764106565.92 total 87.55 last 843.92) - /unlink 55700 (time 1764106575.92 total 97.55 last 726.00) - /unlink 60000 (time 1764106582.89 total 104.53 last 616.87) total: 64768 /unlink in 112.42 seconds: 576.11 ops/second Waiting for MDT destroys to complete oleg411-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 - open/close 3959 (time 1764106617.84 total 10.00 last 395.90) - open/close 8143 (time 1764106627.85 total 20.00 last 418.31) - open/close 10000 (time 1764106631.88 total 24.03 last 460.58) - open/close 14137 (time 1764106641.88 total 34.04 last 413.64) - open/close 18908 (time 1764106651.88 total 44.04 last 477.04) - open/close 20000 (time 1764106654.54 total 46.70 last 410.07) - open/close 24942 (time 1764106664.54 total 56.70 last 494.18) - open/close 29862 (time 1764106674.55 total 66.70 last 491.92) - open/close 34631 (time 1764106684.55 total 76.70 last 476.88) - open/close 38460 (time 1764106694.55 total 86.70 last 382.81) - open/close 40000 (time 1764106697.56 total 89.72 last 511.32) - open/close 45006 (time 1764106707.56 total 99.72 last 500.55) - open/close 49858 (time 1764106717.56 total 109.72 last 485.09) - open/close 54031 (time 1764106727.57 total 119.72 last 417.22) - open/close 58424 (time 1764106737.57 total 129.72 last 439.21) - open/close 60000 (time 1764106741.38 total 133.53 last 413.86) total: 64768 open/close in 142.66 seconds: 454.00 ops/second - /unlink 8298 (time 1764106761.85 total 10.00 last 829.79) - /unlink 10000 (time 1764106763.85 total 12.01 last 848.32) - /unlink 18134 (time 1764106773.85 total 22.01 last 813.32) - /unlink 20000 (time 1764106776.65 total 24.81 last 666.77) - /unlink 26774 (time 1764106786.65 total 34.81 last 677.36) - /unlink 30000 (time 1764106791.32 total 39.47 last 691.75) - /unlink 38340 (time 1764106801.32 total 49.47 last 834.00) - /unlink 46527 (time 1764106811.32 total 59.47 last 818.57) - /unlink 50000 (time 1764106815.62 total 63.77 last 807.90) - /unlink 56343 (time 1764106825.62 total 73.77 last 634.24) - /unlink 60000 (time 1764106831.18 total 79.34 last 657.10) total: 64768 /unlink in 86.33 seconds: 750.23 ops/second Waiting for MDT destroys to complete oleg411-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 - open/close 4673 (time 1764106919.76 total 10.00 last 467.27) - open/close 6660 (time 1764106929.77 total 20.01 last 198.61) - open/close 7841 (time 1764106939.77 total 30.01 last 118.08) - open/close 8763 (time 1764106949.77 total 40.01 last 92.19) - open/close 9637 (time 1764106959.77 total 50.01 last 87.37) - open/close 10000 (time 1764106965.25 total 55.48 last 66.34) - open/close 10626 (time 1764106975.25 total 65.48 last 62.60) - open/close 11311 (time 1764106985.26 total 75.50 last 68.41) - open/close 11937 (time 1764106995.28 total 85.52 last 62.48) - open/close 12575 (time 1764107005.29 total 95.52 last 63.75) - open/close 13392 (time 1764107015.30 total 105.53 last 81.64) - open/close 14315 (time 1764107025.30 total 115.54 last 92.27) - open/close 15269 (time 1764107035.31 total 125.54 last 95.32) - open/close 16243 (time 1764107045.31 total 135.55 last 97.35) - open/close 17247 (time 1764107055.32 total 145.56 last 100.31) - open/close 18195 (time 1764107065.33 total 155.57 last 94.73) - open/close 19191 (time 1764107075.33 total 165.57 last 99.59) - open/close 20000 (time 1764107083.02 total 173.26 last 105.22) - open/close 20962 (time 1764107093.02 total 183.26 last 96.16) - open/close 21981 (time 1764107103.03 total 193.27 last 101.84) - open/close 22937 (time 1764107113.03 total 203.27 last 95.57) - open/close 23993 (time 1764107123.04 total 213.28 last 105.50) - open/close 25035 (time 1764107133.05 total 223.28 last 104.13) - open/close 26048 (time 1764107143.05 total 233.29 last 101.27) - open/close 27048 (time 1764107153.06 total 243.30 last 99.93) - open/close 28065 (time 1764107163.07 total 253.31 last 101.54) - open/close 29002 (time 1764107173.08 total 263.31 last 93.68) - open/close 29993 (time 1764107183.09 total 273.32 last 99.01) - open/close 30912 (time 1764107193.09 total 283.32 last 91.90) - open/close 31843 (time 1764107203.10 total 293.34 last 92.97) - open/close 32845 (time 1764107213.10 total 303.34 last 100.19) - open/close 33929 (time 1764107223.10 total 313.34 last 108.37) - open/close 35087 (time 1764107233.10 total 323.34 last 115.80) - open/close 37398 (time 1764107243.11 total 333.34 last 231.02) - open/close 38684 (time 1764107253.11 total 343.35 last 128.58) - open/close 39757 (time 1764107263.12 total 353.35 last 107.23) - open/close 40000 (time 1764107265.31 total 355.55 last 110.54) - open/close 41311 (time 1764107275.32 total 365.55 last 131.07) - open/close 43187 (time 1764107285.32 total 375.56 last 187.54) - open/close 46259 (time 1764107295.32 total 385.56 last 307.16) - open/close 48966 (time 1764107305.33 total 395.56 last 270.57) - open/close 50000 (time 1764107310.23 total 400.47 last 210.88) - open/close 52053 (time 1764107320.23 total 410.47 last 205.24) - open/close 54245 (time 1764107330.24 total 420.47 last 219.14) - open/close 56010 (time 1764107340.24 total 430.47 last 176.47) - open/close 57233 (time 1764107350.24 total 440.48 last 122.27) - open/close 59359 (time 1764107360.24 total 450.48 last 212.54) - open/close 60000 (time 1764107363.01 total 453.25 last 231.27) - open/close 62762 (time 1764107373.01 total 463.25 last 276.19) total: 64768 open/close in 469.66 seconds: 137.90 ops/second - /unlink 3396 (time 1764107391.20 total 10.00 last 339.60) - /unlink 7362 (time 1764107401.21 total 20.00 last 396.44) - /unlink 10000 (time 1764107406.92 total 25.72 last 461.84) - /unlink 15286 (time 1764107416.92 total 35.72 last 528.50) - /unlink 20000 (time 1764107425.14 total 43.93 last 573.85) - /unlink 26645 (time 1764107435.14 total 53.93 last 664.45) - /unlink 30000 (time 1764107439.29 total 58.08 last 808.63) - /unlink 37924 (time 1764107449.29 total 68.08 last 792.39) - /unlink 40000 (time 1764107454.63 total 73.42 last 388.78) - /unlink 44561 (time 1764107464.63 total 83.43 last 455.97) - /unlink 49674 (time 1764107474.63 total 93.43 last 511.22) - /unlink 55899 (time 1764107484.63 total 103.43 last 622.44) - /unlink 60000 (time 1764107490.27 total 109.07 last 727.41) total: 64768 /unlink in 115.51 seconds: 560.72 ops/second Waiting for MDT destroys to complete oleg411-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 - open/close 4593 (time 1764107540.47 total 10.00 last 459.25) - open/close 9101 (time 1764107550.47 total 20.00 last 450.79) - open/close 13581 (time 1764107560.47 total 30.00 last 447.95) - open/close 18863 (time 1764107570.47 total 40.00 last 528.16) - open/close 20000 (time 1764107572.73 total 42.26 last 503.02) - open/close 24616 (time 1764107582.73 total 52.27 last 461.52) - open/close 29620 (time 1764107592.73 total 62.27 last 500.39) - open/close 34555 (time 1764107602.73 total 72.27 last 493.49) - open/close 38942 (time 1764107612.73 total 82.27 last 438.66) - open/close 40000 (time 1764107614.85 total 84.39 last 499.34) - open/close 44914 (time 1764107624.85 total 94.39 last 491.40) - open/close 50000 (time 1764107634.75 total 104.28 last 513.87) - open/close 54247 (time 1764107644.75 total 114.28 last 424.65) - open/close 58556 (time 1764107654.75 total 124.29 last 430.83) - open/close 60000 (time 1764107657.74 total 127.27 last 483.71) - open/close 64658 (time 1764107667.74 total 137.27 last 465.79) total: 64768 open/close in 137.48 seconds: 471.11 ops/second - /unlink 6849 (time 1764107679.28 total 10.00 last 684.89) - /unlink 10000 (time 1764107683.72 total 14.44 last 709.82) - /unlink 17920 (time 1764107693.72 total 24.44 last 791.91) - /unlink 20000 (time 1764107696.64 total 27.36 last 712.28) - /unlink 28150 (time 1764107706.64 total 37.36 last 814.97) - /unlink 30000 (time 1764107708.67 total 39.38 last 914.25) - /unlink 38299 (time 1764107718.67 total 49.39 last 829.83) - /unlink 46543 (time 1764107728.67 total 59.39 last 824.31) - /unlink 50000 (time 1764107732.64 total 63.36 last 869.31) - /unlink 58538 (time 1764107742.64 total 73.36 last 853.74) total: 64768 /unlink in 81.20 seconds: 797.61 ops/second Waiting for MDT destroys to complete oleg411-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 - open/close 1113 (time 1764107803.47 total 10.01 last 111.17) - open/close 2554 (time 1764107813.47 total 20.02 last 144.01) - open/close 4324 (time 1764107823.48 total 30.02 last 176.96) - open/close 6360 (time 1764107833.48 total 40.02 last 203.59) - open/close 8272 (time 1764107843.48 total 50.03 last 191.10) - open/close 9794 (time 1764107853.49 total 60.03 last 152.15) - open/close 11279 (time 1764107863.49 total 70.03 last 148.48) - open/close 13361 (time 1764107873.49 total 80.03 last 208.13) - open/close 15048 (time 1764107883.50 total 90.04 last 168.63) - open/close 16019 (time 1764107893.50 total 100.04 last 97.04) - open/close 16922 (time 1764107903.51 total 110.06 last 90.20) - open/close 17721 (time 1764107913.51 total 120.06 last 79.89) - open/close 18551 (time 1764107923.51 total 130.06 last 82.99) - open/close 19460 (time 1764107933.54 total 140.08 last 90.68) - open/close 20000 (time 1764107938.25 total 144.79 last 114.60) - open/close 21184 (time 1764107948.26 total 154.80 last 118.33) - open/close 22306 (time 1764107958.26 total 164.81 last 112.11) - open/close 23434 (time 1764107968.27 total 174.81 last 112.74) - open/close 24521 (time 1764107978.28 total 184.82 last 108.58) - open/close 25481 (time 1764107988.28 total 194.82 last 96.00) - open/close 26433 (time 1764107998.29 total 204.83 last 95.16) - open/close 27349 (time 1764108008.29 total 214.83 last 91.58) - open/close 28271 (time 1764108018.29 total 224.84 last 92.15) - open/close 29192 (time 1764108028.29 total 234.84 last 92.10) - open/close 30000 (time 1764108036.76 total 243.31 last 95.40) - open/close 30936 (time 1764108046.77 total 253.31 last 93.57) - open/close 31763 (time 1764108056.77 total 263.31 last 82.67) - open/close 32318 (time 1764108066.78 total 273.32 last 55.45) - open/close 32905 (time 1764108076.78 total 283.33 last 58.67) - open/close 33566 (time 1764108086.79 total 293.33 last 66.06) - open/close 34379 (time 1764108096.79 total 303.33 last 81.29) - open/close 35230 (time 1764108106.80 total 313.34 last 85.04) - open/close 36133 (time 1764108116.81 total 323.35 last 90.24) - open/close 37052 (time 1764108126.82 total 333.36 last 91.80) - open/close 38010 (time 1764108136.82 total 343.36 last 95.76) - open/close 38874 (time 1764108146.83 total 353.37 last 86.31) - open/close 39783 (time 1764108156.84 total 363.38 last 90.83) - open/close 40000 (time 1764108159.31 total 365.85 last 87.90) - open/close 40920 (time 1764108169.31 total 375.85 last 91.96) - open/close 41745 (time 1764108179.31 total 385.85 last 82.50) - open/close 42551 (time 1764108189.32 total 395.86 last 80.54) - open/close 43304 (time 1764108199.32 total 405.86 last 75.28) - open/close 44062 (time 1764108209.33 total 415.87 last 75.78) - open/close 44926 (time 1764108219.33 total 425.87 last 86.34) - open/close 45825 (time 1764108229.33 total 435.88 last 89.88) - open/close 46663 (time 1764108239.35 total 445.89 last 83.66) - open/close 47504 (time 1764108249.37 total 455.91 last 83.92) - open/close 48414 (time 1764108259.38 total 465.92 last 90.91) - open/close 49246 (time 1764108269.40 total 475.94 last 83.07) - open/close 50000 (time 1764108278.00 total 484.55 last 87.59) - open/close 50831 (time 1764108288.01 total 494.55 last 83.08) - open/close 51651 (time 1764108298.02 total 504.56 last 81.93) - open/close 52522 (time 1764108308.04 total 514.58 last 86.93) - open/close 53353 (time 1764108318.04 total 524.59 last 83.03) - open/close 54193 (time 1764108328.06 total 534.60 last 83.89) - open/close 55063 (time 1764108338.06 total 544.60 last 86.96) - open/close 55914 (time 1764108348.06 total 554.61 last 85.07) - open/close 56732 (time 1764108358.07 total 564.61 last 81.78) - open/close 57564 (time 1764108368.07 total 574.61 last 83.19) - open/close 58375 (time 1764108378.08 total 584.62 last 81.05) - open/close 59172 (time 1764108388.09 total 594.63 last 79.60) - open/close 60000 (time 1764108397.49 total 604.03 last 88.04) - open/close 60871 (time 1764108407.50 total 614.04 last 87.04) - open/close 61667 (time 1764108417.51 total 624.06 last 79.48) - open/close 62456 (time 1764108427.52 total 634.06 last 78.88) - open/close 63238 (time 1764108437.53 total 644.08 last 78.08) - open/close 64044 (time 1764108447.54 total 654.09 last 80.51) total: 64768 open/close in 662.53 seconds: 97.76 ops/second - /unlink 798 (time 1764108469.56 total 10.01 last 79.71) - /unlink 1681 (time 1764108479.58 total 20.02 last 88.18) - /unlink 2579 (time 1764108489.58 total 30.03 last 89.77) - /unlink 3505 (time 1764108499.59 total 40.03 last 92.54) - /unlink 4410 (time 1764108509.59 total 50.04 last 90.45) - /unlink 5283 (time 1764108519.60 total 60.05 last 87.23) - /unlink 6205 (time 1764108529.61 total 70.06 last 92.13) - /unlink 7139 (time 1764108539.61 total 80.06 last 93.34) - /unlink 8001 (time 1764108549.62 total 90.07 last 86.11) - /unlink 8608 (time 1764108559.63 total 100.08 last 60.68) - /unlink 9151 (time 1764108569.64 total 110.09 last 54.20) - /unlink 9659 (time 1764108579.66 total 120.11 last 50.71) - /unlink 10000 (time 1764108585.16 total 125.61 last 61.97) - /unlink 10840 (time 1764108595.17 total 135.61 last 83.99) - /unlink 11719 (time 1764108605.17 total 145.62 last 87.84) - /unlink 12546 (time 1764108615.17 total 155.62 last 82.70) - /unlink 13456 (time 1764108625.18 total 165.63 last 90.89) - /unlink 14250 (time 1764108635.19 total 175.64 last 79.32) - /unlink 14948 (time 1764108645.21 total 185.66 last 69.70) - /unlink 15635 (time 1764108655.22 total 195.67 last 68.61) - /unlink 16386 (time 1764108665.23 total 205.68 last 75.06) - /unlink 17186 (time 1764108675.23 total 215.68 last 79.97) - /unlink 17980 (time 1764108685.25 total 225.70 last 79.27) - /unlink 18690 (time 1764108695.26 total 235.71 last 70.93) - /unlink 19406 (time 1764108705.26 total 245.71 last 71.55) - /unlink 20000 (time 1764108713.28 total 253.72 last 74.15) - /unlink 20758 (time 1764108723.29 total 263.74 last 75.69) - /unlink 21565 (time 1764108733.30 total 273.75 last 80.58) - /unlink 22251 (time 1764108743.31 total 283.75 last 68.59) - /unlink 22999 (time 1764108753.32 total 293.77 last 74.71) - /unlink 23738 (time 1764108763.34 total 303.78 last 73.78) - /unlink 24725 (time 1764108773.35 total 313.80 last 98.56) - /unlink 25509 (time 1764108783.35 total 323.80 last 78.37) - /unlink 26255 (time 1764108793.36 total 333.81 last 74.57) - /unlink 27018 (time 1764108803.36 total 343.81 last 76.26) - /unlink 27841 (time 1764108813.36 total 353.81 last 82.28) - /unlink 28692 (time 1764108823.38 total 363.83 last 84.98) - /unlink 29373 (time 1764108833.39 total 373.84 last 68.01) - /unlink 30000 (time 1764108843.15 total 383.60 last 64.27) - /unlink 30679 (time 1764108853.15 total 393.60 last 67.88) - /unlink 31473 (time 1764108863.15 total 403.60 last 79.40) - /unlink 32186 (time 1764108873.16 total 413.61 last 71.24) - /unlink 32912 (time 1764108883.16 total 423.61 last 72.56) - /unlink 33584 (time 1764108893.16 total 433.61 last 67.19) - /unlink 34349 (time 1764108903.18 total 443.62 last 76.41) - /unlink 35181 (time 1764108913.18 total 453.63 last 83.18) - /unlink 35893 (time 1764108923.18 total 463.63 last 71.19) - /unlink 36626 (time 1764108933.18 total 473.63 last 73.28) - /unlink 37411 (time 1764108943.19 total 483.64 last 78.42) - /unlink 38153 (time 1764108953.20 total 493.65 last 74.13) - /unlink 38965 (time 1764108963.21 total 503.65 last 81.17) - /unlink 39711 (time 1764108973.21 total 513.65 last 74.59) - /unlink 40000 (time 1764108977.01 total 517.46 last 75.93) - /unlink 40764 (time 1764108987.02 total 527.47 last 76.35) - /unlink 41546 (time 1764108997.03 total 537.48 last 78.12) - /unlink 42374 (time 1764109007.04 total 547.49 last 82.74) - /unlink 43201 (time 1764109017.04 total 557.49 last 82.64) - /unlink 43941 (time 1764109027.05 total 567.49 last 73.99) - /unlink 44806 (time 1764109037.05 total 577.50 last 86.44) - /unlink 45590 (time 1764109047.06 total 587.50 last 78.38) - /unlink 46267 (time 1764109057.06 total 597.51 last 67.66) - /unlink 46904 (time 1764109067.06 total 607.51 last 63.68) - /unlink 47565 (time 1764109077.07 total 617.52 last 66.04) - /unlink 48164 (time 1764109087.09 total 627.54 last 59.81) - /unlink 48905 (time 1764109097.09 total 637.54 last 74.10) - /unlink 49632 (time 1764109107.10 total 647.55 last 72.59) - /unlink 50000 (time 1764109112.75 total 653.20 last 65.13) - /unlink 50840 (time 1764109122.76 total 663.21 last 83.93) - /unlink 51743 (time 1764109132.79 total 673.24 last 90.05) - /unlink 52596 (time 1764109142.82 total 683.27 last 85.05) - /unlink 53448 (time 1764109152.82 total 693.27 last 85.19) - /unlink 54307 (time 1764109162.85 total 703.30 last 85.66) - /unlink 55215 (time 1764109172.85 total 713.30 last 90.76) - /unlink 56058 (time 1764109182.87 total 723.32 last 84.16) - /unlink 56972 (time 1764109192.89 total 733.34 last 91.22) - /unlink 57920 (time 1764109202.92 total 743.37 last 94.53) - /unlink 58740 (time 1764109212.94 total 753.39 last 81.80) - /unlink 59637 (time 1764109222.97 total 763.41 last 89.51) - /unlink 60000 (time 1764109226.84 total 767.29 last 93.60) - /unlink 61106 (time 1764109236.85 total 777.30 last 110.56) - /unlink 62117 (time 1764109246.85 total 787.30 last 101.08) - /unlink 63105 (time 1764109256.86 total 797.31 last 98.71) - /unlink 64049 (time 1764109266.87 total 807.32 last 94.29) total: 64768 /unlink in 815.06 seconds: 79.46 ops/second Waiting for MDT destroys to complete oleg411-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing unload_modules_local oleg411-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg411-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 PASS 106 (3202s) Stopping clients: oleg411-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg411-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 oleg411-server: oleg411-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg411-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3348 sec ========== 17:22:39 (1764109359) === conf-sanity: start cleanup 17:22:40 (1764109360) === === conf-sanity: finish cleanup 17:22:42 (1764109362) ===