-----============= acceptance-small: conf-sanity ============----- Tue Apr 14 13:26:49 EDT 2026 oleg360-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg360-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg360-server oleg360-server: oleg360-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg360-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg360-server: detected 4 online CPUs by sysfs oleg360-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg360-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg360-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg360-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg360-server: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg360-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg360-server: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg360-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:30:00 (1776187800) Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:-f) oleg360-server: oleg360-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg360-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg360-server: detected 4 online CPUs by sysfs oleg360-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg360-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg360-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg360-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg360-client.virtnet: -o user_xattr,flock oleg360-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 730 (time 1776187923.52 total 10.01 last 72.94) - open/close 1383 (time 1776187933.53 total 20.02 last 65.20) - open/close 2177 (time 1776187943.54 total 30.03 last 79.33) - open/close 2993 (time 1776187953.55 total 40.04 last 81.54) - open/close 3741 (time 1776187963.55 total 50.05 last 74.76) - open/close 4466 (time 1776187973.57 total 60.06 last 72.42) - open/close 5220 (time 1776187983.58 total 70.07 last 75.30) - open/close 6025 (time 1776187993.59 total 80.08 last 80.41) - open/close 6840 (time 1776188003.60 total 90.09 last 81.46) - open/close 7686 (time 1776188013.60 total 100.09 last 84.54) - open/close 8423 (time 1776188023.60 total 110.10 last 73.68) - open/close 9188 (time 1776188033.64 total 120.13 last 76.25) - open/close 9924 (time 1776188043.64 total 130.13 last 73.59) - open/close 10627 (time 1776188053.64 total 140.13 last 70.30) - open/close 11416 (time 1776188063.64 total 150.13 last 78.88) - open/close 12048 (time 1776188073.65 total 160.14 last 63.17) - open/close 12857 (time 1776188083.65 total 170.14 last 80.88) - open/close 13592 (time 1776188093.66 total 180.16 last 73.40) - open/close 14297 (time 1776188103.67 total 190.17 last 70.43) - open/close 15032 (time 1776188113.67 total 200.17 last 73.49) - open/close 15775 (time 1776188123.68 total 210.18 last 74.23) - open/close 16398 (time 1776188133.70 total 220.19 last 62.23) - open/close 16924 (time 1776188143.70 total 230.19 last 52.58) - open/close 17690 (time 1776188153.72 total 240.21 last 76.47) - open/close 18469 (time 1776188163.72 total 250.21 last 77.85) - open/close 19227 (time 1776188173.73 total 260.22 last 75.73) - open/close 20000 (time 1776188183.62 total 270.11 last 78.15) - open/close 20804 (time 1776188193.62 total 280.11 last 80.39) - open/close 21643 (time 1776188203.62 total 290.12 last 83.89) - open/close 22398 (time 1776188213.64 total 300.13 last 75.39) - open/close 23153 (time 1776188223.65 total 310.14 last 75.44) - open/close 23874 (time 1776188233.65 total 320.15 last 72.06) - open/close 24625 (time 1776188243.67 total 330.16 last 74.97) - open/close 25399 (time 1776188253.68 total 340.17 last 77.33) - open/close 26169 (time 1776188263.69 total 350.19 last 76.90) - open/close 26874 (time 1776188273.70 total 360.19 last 70.49) - open/close 27638 (time 1776188283.71 total 370.20 last 76.32) - open/close 28376 (time 1776188293.71 total 380.21 last 73.74) - open/close 29096 (time 1776188303.72 total 390.21 last 71.95) - open/close 29852 (time 1776188313.72 total 400.21 last 75.59) - open/close 30000 (time 1776188315.98 total 402.47 last 65.71) - open/close 30708 (time 1776188325.98 total 412.47 last 70.78) - open/close 31413 (time 1776188335.98 total 422.47 last 70.49) - open/close 32080 (time 1776188345.98 total 432.47 last 66.68) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 440.82 seconds: 73.94 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1251 (time 1776188369.03 total 10.00 last 125.09) - /unlink 2424 (time 1776188379.03 total 20.00 last 117.29) - /unlink 3373 (time 1776188389.05 total 30.02 last 94.75) - /unlink 4698 (time 1776188399.06 total 40.03 last 132.37) - /unlink 5955 (time 1776188409.07 total 50.04 last 125.56) - /unlink 7254 (time 1776188419.08 total 60.04 last 129.83) - /unlink 8707 (time 1776188429.08 total 70.05 last 145.25) - /unlink 10000 (time 1776188438.16 total 79.13 last 142.36) - /unlink 11411 (time 1776188448.17 total 89.14 last 140.97) - /unlink 12842 (time 1776188458.18 total 99.14 last 143.05) - /unlink 14161 (time 1776188468.18 total 109.15 last 131.84) - /unlink 15328 (time 1776188478.18 total 119.15 last 116.65) - /unlink 16509 (time 1776188488.19 total 129.16 last 118.04) - /unlink 17582 (time 1776188498.19 total 139.16 last 107.29) - /unlink 18603 (time 1776188508.20 total 149.17 last 102.02) - /unlink 19910 (time 1776188518.20 total 159.17 last 130.63) - /unlink 21223 (time 1776188528.21 total 169.18 last 131.26) - /unlink 22463 (time 1776188538.21 total 179.18 last 123.94) - /unlink 23748 (time 1776188548.21 total 189.18 last 128.48) - /unlink 25101 (time 1776188558.22 total 199.19 last 135.19) - /unlink 26373 (time 1776188568.24 total 209.21 last 126.98) - /unlink 27747 (time 1776188578.24 total 219.21 last 137.36) - /unlink 29021 (time 1776188588.24 total 229.21 last 127.38) - /unlink 30000 (time 1776188595.49 total 236.46 last 135.14) - /unlink 31277 (time 1776188605.49 total 246.46 last 127.66) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 256.25 seconds: 127.20 ops/second Waiting for MDT destroys to complete - open/close 787 (time 1776188644.92 total 10.00 last 78.68) - open/close 1520 (time 1776188654.93 total 20.01 last 73.23) - open/close 2274 (time 1776188664.94 total 30.02 last 75.31) - open/close 3024 (time 1776188674.94 total 40.03 last 74.98) - open/close 3810 (time 1776188684.94 total 50.03 last 78.59) - open/close 4576 (time 1776188694.96 total 60.05 last 76.45) - open/close 5392 (time 1776188704.96 total 70.05 last 81.60) - open/close 6148 (time 1776188714.97 total 80.05 last 75.55) - open/close 6981 (time 1776188724.97 total 90.05 last 83.29) - open/close 7756 (time 1776188734.97 total 100.06 last 77.49) - open/close 8517 (time 1776188744.98 total 110.06 last 76.06) - open/close 9264 (time 1776188754.99 total 120.07 last 74.63) - open/close 10000 (time 1776188764.85 total 129.93 last 74.65) - open/close 10781 (time 1776188774.86 total 139.95 last 77.96) - open/close 11535 (time 1776188784.88 total 149.96 last 75.31) - open/close 12317 (time 1776188794.89 total 159.97 last 78.11) - open/close 13151 (time 1776188804.89 total 169.97 last 83.40) - open/close 13864 (time 1776188814.90 total 179.98 last 71.22) - open/close 14667 (time 1776188824.91 total 189.99 last 80.24) - open/close 15423 (time 1776188834.91 total 200.00 last 75.57) - open/close 16188 (time 1776188844.92 total 210.00 last 76.47) - open/close 16966 (time 1776188854.92 total 220.00 last 77.77) - open/close 17759 (time 1776188864.93 total 230.01 last 79.25) - open/close 18584 (time 1776188874.93 total 240.01 last 82.47) - open/close 19311 (time 1776188884.93 total 250.02 last 72.68) - open/close 20000 (time 1776188894.43 total 259.51 last 72.55) - open/close 20726 (time 1776188904.43 total 269.51 last 72.59) - open/close 21469 (time 1776188914.44 total 279.52 last 74.24) - open/close 22224 (time 1776188924.45 total 289.53 last 75.42) - open/close 22978 (time 1776188934.46 total 299.54 last 75.35) - open/close 23745 (time 1776188944.46 total 309.55 last 76.65) - open/close 24480 (time 1776188954.47 total 319.56 last 73.42) - open/close 25243 (time 1776188964.48 total 329.56 last 76.25) - open/close 26055 (time 1776188974.49 total 339.57 last 81.15) - open/close 26808 (time 1776188984.49 total 349.57 last 75.30) - open/close 27568 (time 1776188994.49 total 359.57 last 75.96) - open/close 28298 (time 1776189004.50 total 369.59 last 72.91) - open/close 29049 (time 1776189014.51 total 379.59 last 75.05) - open/close 29742 (time 1776189024.54 total 389.62 last 69.11) - open/close 30000 (time 1776189027.97 total 393.06 last 75.08) - open/close 30756 (time 1776189037.98 total 403.06 last 75.56) - open/close 31431 (time 1776189047.99 total 413.07 last 67.44) - open/close 32157 (time 1776189057.99 total 423.08 last 72.56) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 429.20 seconds: 75.94 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1233 (time 1776189078.20 total 10.01 last 123.24) - /unlink 2421 (time 1776189088.21 total 20.01 last 118.76) - /unlink 3744 (time 1776189098.21 total 30.01 last 132.29) - /unlink 4983 (time 1776189108.22 total 40.03 last 123.69) - /unlink 6306 (time 1776189118.23 total 50.03 last 132.20) - /unlink 7554 (time 1776189128.24 total 60.04 last 124.74) - /unlink 8741 (time 1776189138.24 total 70.04 last 118.66) - /unlink 9943 (time 1776189148.25 total 80.05 last 120.13) - /unlink 11153 (time 1776189158.25 total 90.05 last 120.96) - /unlink 12422 (time 1776189168.26 total 100.06 last 126.77) - /unlink 13670 (time 1776189178.27 total 110.07 last 124.67) - /unlink 14976 (time 1776189188.27 total 120.07 last 130.57) - /unlink 16210 (time 1776189198.27 total 130.08 last 123.38) - /unlink 17572 (time 1776189208.28 total 140.09 last 136.06) - /unlink 18900 (time 1776189218.29 total 150.09 last 132.75) - /unlink 20000 (time 1776189226.19 total 157.99 last 139.16) - /unlink 21358 (time 1776189236.19 total 168.00 last 135.78) - /unlink 22703 (time 1776189246.20 total 178.00 last 134.45) - /unlink 23926 (time 1776189256.21 total 188.01 last 122.18) - /unlink 25206 (time 1776189266.21 total 198.01 last 127.96) - /unlink 26583 (time 1776189276.21 total 208.01 last 137.68) - /unlink 27953 (time 1776189286.21 total 218.02 last 136.98) - /unlink 29443 (time 1776189296.22 total 228.02 last 148.93) - /unlink 30000 (time 1776189299.76 total 231.57 last 157.05) - /unlink 31372 (time 1776189309.77 total 241.57 last 137.11) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 250.92 seconds: 129.90 ops/second Waiting for MDT destroys to complete - open/close 774 (time 1776189342.15 total 10.01 last 77.32) - open/close 1623 (time 1776189352.15 total 20.01 last 84.88) - open/close 2383 (time 1776189362.16 total 30.02 last 75.93) - open/close 3123 (time 1776189372.16 total 40.02 last 74.00) - open/close 3869 (time 1776189382.17 total 50.03 last 74.51) - open/close 4593 (time 1776189392.18 total 60.04 last 72.33) - open/close 5241 (time 1776189402.18 total 70.05 last 64.79) - open/close 5956 (time 1776189412.19 total 80.05 last 71.47) - open/close 6732 (time 1776189422.19 total 90.05 last 77.57) - open/close 7516 (time 1776189432.20 total 100.06 last 78.36) - open/close 8269 (time 1776189442.21 total 110.07 last 75.22) - open/close 8972 (time 1776189452.21 total 120.07 last 70.28) - open/close 9721 (time 1776189462.22 total 130.08 last 74.82) - open/close 10000 (time 1776189466.15 total 134.02 last 70.92) - open/close 10769 (time 1776189476.16 total 144.02 last 76.86) - open/close 11508 (time 1776189486.17 total 154.03 last 73.82) - open/close 12242 (time 1776189496.17 total 164.04 last 73.37) - open/close 12958 (time 1776189506.18 total 174.05 last 71.52) - open/close 13658 (time 1776189516.21 total 184.08 last 69.80) - open/close 14327 (time 1776189526.22 total 194.08 last 66.87) - open/close 15053 (time 1776189536.23 total 204.09 last 72.52) - open/close 15768 (time 1776189546.24 total 214.10 last 71.43) - open/close 16444 (time 1776189556.24 total 224.10 last 67.59) - open/close 17185 (time 1776189566.24 total 234.11 last 74.08) - open/close 17885 (time 1776189576.25 total 244.11 last 69.98) - open/close 18658 (time 1776189586.25 total 254.12 last 77.25) - open/close 19248 (time 1776189596.26 total 264.12 last 58.96) - open/close 19953 (time 1776189606.27 total 274.13 last 70.42) - open/close 20652 (time 1776189616.27 total 284.14 last 69.89) - open/close 21317 (time 1776189626.28 total 294.14 last 66.48) - open/close 22049 (time 1776189636.29 total 304.16 last 73.07) - open/close 22735 (time 1776189646.30 total 314.16 last 68.55) - open/close 23414 (time 1776189656.31 total 324.17 last 67.85) - open/close 24087 (time 1776189666.31 total 334.17 last 67.30) - open/close 24784 (time 1776189676.33 total 344.19 last 69.55) - open/close 25482 (time 1776189686.34 total 354.21 last 69.71) - open/close 26212 (time 1776189696.35 total 364.21 last 72.95) - open/close 26890 (time 1776189706.35 total 374.21 last 67.79) - open/close 27587 (time 1776189716.35 total 384.21 last 69.70) - open/close 28223 (time 1776189726.36 total 394.22 last 63.55) - open/close 28701 (time 1776189736.36 total 404.22 last 47.80) - open/close 29232 (time 1776189746.37 total 414.23 last 53.06) - open/close 29769 (time 1776189756.39 total 424.25 last 53.60) - open/close 30000 (time 1776189759.92 total 427.78 last 65.36) - open/close 30703 (time 1776189769.94 total 437.80 last 70.15) - open/close 31340 (time 1776189779.95 total 447.81 last 63.64) - open/close 31998 (time 1776189789.96 total 457.82 last 65.78) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 467.14 seconds: 69.77 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1163 (time 1776189814.14 total 10.01 last 116.24) - /unlink 2392 (time 1776189824.14 total 20.01 last 122.87) - /unlink 3557 (time 1776189834.15 total 30.02 last 116.38) - /unlink 4704 (time 1776189844.15 total 40.02 last 114.69) - /unlink 5808 (time 1776189854.15 total 50.02 last 110.34) - /unlink 7034 (time 1776189864.16 total 60.03 last 122.58) - /unlink 8230 (time 1776189874.17 total 70.04 last 119.48) - /unlink 9551 (time 1776189884.17 total 80.04 last 132.09) - /unlink 10000 (time 1776189888.35 total 84.22 last 107.22) - /unlink 11216 (time 1776189898.36 total 94.23 last 121.56) - /unlink 12383 (time 1776189908.36 total 104.23 last 116.69) - /unlink 13561 (time 1776189918.36 total 114.23 last 117.73) - /unlink 14789 (time 1776189928.37 total 124.23 last 122.79) - /unlink 15808 (time 1776189938.37 total 134.24 last 101.88) - /unlink 17026 (time 1776189948.37 total 144.24 last 121.72) - /unlink 18241 (time 1776189958.38 total 154.25 last 121.46) - /unlink 19581 (time 1776189968.38 total 164.25 last 133.94) - /unlink 20000 (time 1776189971.72 total 167.59 last 125.47) - /unlink 21307 (time 1776189981.73 total 177.60 last 130.65) - /unlink 22563 (time 1776189991.73 total 187.60 last 125.59) - /unlink 23898 (time 1776190001.73 total 197.60 last 133.50) - /unlink 25187 (time 1776190011.73 total 207.60 last 128.84) - /unlink 26429 (time 1776190021.74 total 217.61 last 124.13) - /unlink 27706 (time 1776190031.74 total 227.61 last 127.66) - /unlink 29019 (time 1776190041.74 total 237.61 last 131.24) - /unlink 30000 (time 1776190049.49 total 245.36 last 126.62) - /unlink 31258 (time 1776190059.50 total 255.37 last 125.71) - /unlink 32560 (time 1776190069.50 total 265.37 last 130.13) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 265.63 seconds: 122.70 ops/second Waiting for MDT destroys to complete - open/close 774 (time 1776190094.40 total 10.01 last 77.29) - open/close 1556 (time 1776190104.42 total 20.03 last 78.06) - open/close 2326 (time 1776190114.42 total 30.03 last 76.99) - open/close 3061 (time 1776190124.43 total 40.04 last 73.42) - open/close 3832 (time 1776190134.45 total 50.06 last 76.96) - open/close 4650 (time 1776190144.46 total 60.07 last 81.73) - open/close 5347 (time 1776190154.46 total 70.08 last 69.67) - open/close 6143 (time 1776190164.46 total 80.08 last 79.57) - open/close 6904 (time 1776190174.47 total 90.08 last 76.07) - open/close 7659 (time 1776190184.48 total 100.10 last 75.40) - open/close 8478 (time 1776190194.48 total 110.10 last 81.90) - open/close 9220 (time 1776190204.49 total 120.10 last 74.16) - open/close 10000 (time 1776190213.96 total 129.58 last 82.32) - open/close 10849 (time 1776190223.97 total 139.59 last 84.83) - open/close 11690 (time 1776190233.98 total 149.60 last 84.02) - open/close 12561 (time 1776190243.99 total 159.61 last 87.01) - open/close 13395 (time 1776190254.00 total 169.61 last 83.34) - open/close 14308 (time 1776190264.01 total 179.63 last 91.17) - open/close 15080 (time 1776190274.03 total 189.64 last 77.11) - open/close 15718 (time 1776190284.03 total 199.65 last 63.75) - open/close 16427 (time 1776190294.03 total 209.65 last 70.90) - open/close 17166 (time 1776190304.04 total 219.65 last 73.88) - open/close 17825 (time 1776190314.04 total 229.66 last 65.85) - open/close 18497 (time 1776190324.05 total 239.67 last 67.15) - open/close 19218 (time 1776190334.06 total 249.68 last 72.01) - open/close 20000 (time 1776190343.79 total 259.40 last 80.40) - open/close 20757 (time 1776190353.81 total 269.42 last 75.57) - open/close 21550 (time 1776190363.81 total 279.42 last 79.29) - open/close 22410 (time 1776190373.81 total 289.43 last 85.94) - open/close 23291 (time 1776190383.83 total 299.44 last 88.01) - open/close 24215 (time 1776190393.83 total 309.44 last 92.37) - open/close 25041 (time 1776190403.83 total 319.45 last 82.56) - open/close 25875 (time 1776190413.85 total 329.47 last 83.23) - open/close 26526 (time 1776190423.86 total 339.48 last 65.05) - open/close 27212 (time 1776190433.87 total 349.48 last 68.55) - open/close 27849 (time 1776190443.88 total 359.49 last 63.66) - open/close 28549 (time 1776190453.88 total 369.50 last 69.95) - open/close 29252 (time 1776190463.89 total 379.51 last 70.22) - open/close 30000 (time 1776190473.90 total 389.51 last 74.78) - open/close 30705 (time 1776190483.91 total 399.52 last 70.42) - open/close 31420 (time 1776190493.92 total 409.53 last 71.41) - open/close 32195 (time 1776190503.93 total 419.54 last 77.47) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 424.58 seconds: 76.77 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1361 (time 1776190523.06 total 10.00 last 136.08) - /unlink 2634 (time 1776190533.06 total 20.01 last 127.24) - /unlink 3720 (time 1776190543.08 total 30.02 last 108.46) - /unlink 4831 (time 1776190553.09 total 40.03 last 110.94) - /unlink 6072 (time 1776190563.10 total 50.04 last 124.03) - /unlink 7401 (time 1776190573.10 total 60.04 last 132.87) - /unlink 8706 (time 1776190583.11 total 70.05 last 130.41) - /unlink 9816 (time 1776190593.11 total 80.05 last 110.97) - /unlink 11149 (time 1776190603.11 total 90.06 last 133.24) - /unlink 12554 (time 1776190613.11 total 100.06 last 140.49) - /unlink 14083 (time 1776190623.12 total 110.06 last 152.81) - /unlink 15418 (time 1776190633.14 total 120.08 last 133.26) - /unlink 16808 (time 1776190643.14 total 130.08 last 138.94) - /unlink 18253 (time 1776190653.14 total 140.09 last 144.48) - /unlink 19507 (time 1776190663.15 total 150.09 last 125.38) - /unlink 20000 (time 1776190666.92 total 153.86 last 130.51) - /unlink 21453 (time 1776190676.92 total 163.86 last 145.29) - /unlink 22857 (time 1776190686.93 total 173.87 last 140.34) - /unlink 24335 (time 1776190696.94 total 183.88 last 147.67) - /unlink 25677 (time 1776190706.96 total 193.90 last 133.94) - /unlink 27035 (time 1776190716.96 total 203.90 last 135.79) - /unlink 28347 (time 1776190726.96 total 213.90 last 131.18) - /unlink 29635 (time 1776190736.96 total 223.90 last 128.79) - /unlink 30000 (time 1776190739.87 total 226.81 last 125.56) - /unlink 31280 (time 1776190749.87 total 236.81 last 128.00) - /unlink 32511 (time 1776190759.87 total 246.82 last 122.99) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 247.35 seconds: 131.77 ops/second Waiting for MDT destroys to complete - open/close 756 (time 1776190786.42 total 10.01 last 75.51) - open/close 1509 (time 1776190796.42 total 20.01 last 75.29) - open/close 2340 (time 1776190806.43 total 30.02 last 83.03) - open/close 3073 (time 1776190816.44 total 40.03 last 73.22) - open/close 3825 (time 1776190826.44 total 50.03 last 75.19) - open/close 4603 (time 1776190836.44 total 60.04 last 77.78) - open/close 5483 (time 1776190846.44 total 70.04 last 88.00) - open/close 6302 (time 1776190856.45 total 80.04 last 81.87) - open/close 7062 (time 1776190866.45 total 90.05 last 75.96) - open/close 7769 (time 1776190876.47 total 100.06 last 70.58) - open/close 8487 (time 1776190886.48 total 110.08 last 71.71) - open/close 9179 (time 1776190896.50 total 120.09 last 69.11) - open/close 9930 (time 1776190906.50 total 130.09 last 75.07) - open/close 10670 (time 1776190916.51 total 140.10 last 73.94) - open/close 11390 (time 1776190926.51 total 150.10 last 71.98) - open/close 12129 (time 1776190936.52 total 160.11 last 73.84) - open/close 12824 (time 1776190946.53 total 170.13 last 69.40) - open/close 13629 (time 1776190956.54 total 180.13 last 80.46) - open/close 14402 (time 1776190966.54 total 190.14 last 77.23) - open/close 15157 (time 1776190976.56 total 200.15 last 75.41) - open/close 15805 (time 1776190986.61 total 210.21 last 64.43) - open/close 16537 (time 1776190996.62 total 220.22 last 73.15) - open/close 17271 (time 1776191006.64 total 230.23 last 73.27) - open/close 17988 (time 1776191016.64 total 240.23 last 71.69) - open/close 18742 (time 1776191026.64 total 250.24 last 75.36) - open/close 19511 (time 1776191036.65 total 260.25 last 76.83) - open/close 20000 (time 1776191042.88 total 266.47 last 78.56) - open/close 20805 (time 1776191052.88 total 276.48 last 80.46) - open/close 21737 (time 1776191062.90 total 286.49 last 93.05) - open/close 22582 (time 1776191072.91 total 296.50 last 84.43) - open/close 23456 (time 1776191082.91 total 306.51 last 87.34) - open/close 24239 (time 1776191092.92 total 316.51 last 78.28) - open/close 25028 (time 1776191102.92 total 326.51 last 78.87) - open/close 25756 (time 1776191112.92 total 336.51 last 72.80) - open/close 26556 (time 1776191122.93 total 346.52 last 79.94) - open/close 27313 (time 1776191132.93 total 356.53 last 75.67) - open/close 28135 (time 1776191142.95 total 366.54 last 82.07) - open/close 29010 (time 1776191152.96 total 376.55 last 87.41) - open/close 29778 (time 1776191162.98 total 386.57 last 76.66) - open/close 30000 (time 1776191165.80 total 389.39 last 78.74) - open/close 30793 (time 1776191175.80 total 399.40 last 79.25) - open/close 31627 (time 1776191185.81 total 409.41 last 83.31) - open/close 32251 (time 1776191195.81 total 419.41 last 62.40) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 425.30 seconds: 76.64 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1251 (time 1776191216.37 total 10.00 last 125.06) - /unlink 2406 (time 1776191226.38 total 20.01 last 115.40) - /unlink 3685 (time 1776191236.38 total 30.02 last 127.86) - /unlink 4877 (time 1776191246.38 total 40.02 last 119.18) - /unlink 5959 (time 1776191256.38 total 50.02 last 108.19) - /unlink 7041 (time 1776191266.38 total 60.02 last 108.19) - /unlink 8195 (time 1776191276.39 total 70.02 last 115.38) - /unlink 9327 (time 1776191286.39 total 80.02 last 113.18) - /unlink 10000 (time 1776191292.11 total 85.74 last 117.71) - /unlink 11267 (time 1776191302.11 total 95.74 last 126.69) - /unlink 12556 (time 1776191312.11 total 105.74 last 128.86) - /unlink 13858 (time 1776191322.11 total 115.75 last 130.14) - /unlink 15311 (time 1776191332.12 total 125.75 last 145.26) - /unlink 16787 (time 1776191342.12 total 135.76 last 147.54) - /unlink 18154 (time 1776191352.12 total 145.76 last 136.66) - /unlink 19505 (time 1776191362.13 total 155.76 last 135.06) - /unlink 20000 (time 1776191365.82 total 159.45 last 134.14) - /unlink 21323 (time 1776191375.82 total 169.45 last 132.27) - /unlink 22625 (time 1776191385.82 total 179.46 last 130.17) - /unlink 23886 (time 1776191395.83 total 189.46 last 126.05) - /unlink 25280 (time 1776191405.83 total 199.46 last 139.38) - /unlink 26689 (time 1776191415.83 total 209.47 last 140.84) - /unlink 28020 (time 1776191425.84 total 219.47 last 133.04) - /unlink 29268 (time 1776191435.84 total 229.48 last 124.72) - /unlink 30000 (time 1776191441.40 total 235.03 last 131.74) - /unlink 31223 (time 1776191451.40 total 245.04 last 122.23) - /unlink 32428 (time 1776191461.41 total 255.04 last 120.49) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 256.46 seconds: 127.09 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg360-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing unload_modules_local oleg360-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg360-server: LNET ready to unload oleg360-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 PASS 106 (3726s) Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 oleg360-server: oleg360-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg360-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg360-server: detected 4 online CPUs by sysfs oleg360-server: Force libcfs to create 2 CPU partitions oleg360-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg360-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3983 sec ========== 14:33:13 (1776191593)