-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 09:54:26 EDT 2025 oleg216-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg216-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg216-server oleg216-server: oleg216-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg216-server: detected 4 online CPUs by sysfs oleg216-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg216-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg216-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg216-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg216-server: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg216-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg216-server: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg216-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:57:42 (1761314262) Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) oleg216-server: oleg216-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg216-server: detected 4 online CPUs by sysfs oleg216-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg216-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg216-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg216-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock oleg216-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 770 (time 1761314395.66 total 10.00 last 76.96) - open/close 1452 (time 1761314405.68 total 20.02 last 68.08) - open/close 2128 (time 1761314415.69 total 30.04 last 67.49) - open/close 2838 (time 1761314425.70 total 40.04 last 70.97) - open/close 3556 (time 1761314435.70 total 50.05 last 71.77) - open/close 4277 (time 1761314445.71 total 60.05 last 72.06) - open/close 5007 (time 1761314455.71 total 70.06 last 72.94) - open/close 5692 (time 1761314465.73 total 80.07 last 68.41) - open/close 6403 (time 1761314475.73 total 90.08 last 71.06) - open/close 7089 (time 1761314485.74 total 100.08 last 68.57) - open/close 7820 (time 1761314495.74 total 110.09 last 73.08) - open/close 8486 (time 1761314505.75 total 120.10 last 66.52) - open/close 9221 (time 1761314515.76 total 130.11 last 73.43) - open/close 9954 (time 1761314525.79 total 140.13 last 73.12) - open/close 10585 (time 1761314535.80 total 150.15 last 63.01) - open/close 11244 (time 1761314545.80 total 160.15 last 65.89) - open/close 11913 (time 1761314555.81 total 170.16 last 66.84) - open/close 12574 (time 1761314565.82 total 180.17 last 66.03) - open/close 13136 (time 1761314575.83 total 190.18 last 56.16) - open/close 13760 (time 1761314585.84 total 200.19 last 62.33) - open/close 14479 (time 1761314595.84 total 210.19 last 71.90) - open/close 15274 (time 1761314605.86 total 220.20 last 79.38) - open/close 16020 (time 1761314615.86 total 230.20 last 74.60) - open/close 16776 (time 1761314625.87 total 240.22 last 75.49) - open/close 17498 (time 1761314635.88 total 250.22 last 72.16) - open/close 18220 (time 1761314645.88 total 260.23 last 72.19) - open/close 18915 (time 1761314655.88 total 270.23 last 69.48) - open/close 19625 (time 1761314665.89 total 280.24 last 70.95) - open/close 20000 (time 1761314670.70 total 285.05 last 77.93) - open/close 20730 (time 1761314680.70 total 295.05 last 72.97) - open/close 21399 (time 1761314690.71 total 305.06 last 66.84) - open/close 22109 (time 1761314700.72 total 315.07 last 70.95) - open/close 22815 (time 1761314710.72 total 325.07 last 70.56) - open/close 23506 (time 1761314720.73 total 335.07 last 69.09) - open/close 24229 (time 1761314730.74 total 345.09 last 72.21) - open/close 24968 (time 1761314740.75 total 355.10 last 73.83) - open/close 25686 (time 1761314750.75 total 365.10 last 71.76) - open/close 26462 (time 1761314760.76 total 375.11 last 77.56) - open/close 27151 (time 1761314770.77 total 385.11 last 68.85) - open/close 27867 (time 1761314780.78 total 395.12 last 71.53) - open/close 28503 (time 1761314790.79 total 405.14 last 63.50) - open/close 29185 (time 1761314800.81 total 415.16 last 68.10) - open/close 29860 (time 1761314810.82 total 425.17 last 67.40) - open/close 30000 (time 1761314813.00 total 427.34 last 64.43) - open/close 30731 (time 1761314823.00 total 437.35 last 73.05) - open/close 31425 (time 1761314833.01 total 447.35 last 69.38) - open/close 32119 (time 1761314843.01 total 457.36 last 69.35) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 463.75 seconds: 70.28 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1236 (time 1761314863.28 total 10.00 last 123.57) - /unlink 2447 (time 1761314873.30 total 20.02 last 120.91) - /unlink 3613 (time 1761314883.31 total 30.03 last 116.51) - /unlink 4854 (time 1761314893.32 total 40.03 last 123.99) - /unlink 5991 (time 1761314903.32 total 50.04 last 113.68) - /unlink 7118 (time 1761314913.32 total 60.04 last 112.63) - /unlink 8340 (time 1761314923.33 total 70.05 last 122.16) - /unlink 9516 (time 1761314933.33 total 80.05 last 117.55) - /unlink 10000 (time 1761314937.19 total 83.91 last 125.48) - /unlink 11165 (time 1761314947.19 total 93.91 last 116.46) - /unlink 12422 (time 1761314957.20 total 103.92 last 125.58) - /unlink 13691 (time 1761314967.20 total 113.92 last 126.86) - /unlink 14965 (time 1761314977.20 total 123.92 last 127.40) - /unlink 16269 (time 1761314987.21 total 133.93 last 130.34) - /unlink 17544 (time 1761314997.21 total 143.93 last 127.43) - /unlink 18762 (time 1761315007.22 total 153.94 last 121.72) - /unlink 20000 (time 1761315017.01 total 163.73 last 126.50) - /unlink 21204 (time 1761315027.01 total 173.73 last 120.34) - /unlink 22359 (time 1761315037.02 total 183.74 last 115.43) - /unlink 23590 (time 1761315047.02 total 193.74 last 123.06) - /unlink 24712 (time 1761315057.03 total 203.75 last 112.14) - /unlink 25952 (time 1761315067.03 total 213.75 last 124.00) - /unlink 27138 (time 1761315077.03 total 223.75 last 118.55) - /unlink 28395 (time 1761315087.04 total 233.76 last 125.62) - /unlink 29667 (time 1761315097.04 total 243.76 last 127.13) - /unlink 30000 (time 1761315099.65 total 246.37 last 127.80) - /unlink 31296 (time 1761315109.66 total 256.38 last 129.49) - /unlink 32481 (time 1761315119.66 total 266.38 last 118.50) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 267.41 seconds: 121.89 ops/second Waiting for MDT destroys to complete - open/close 763 (time 1761315147.07 total 10.01 last 76.24) - open/close 1452 (time 1761315157.08 total 20.02 last 68.83) - open/close 2188 (time 1761315167.09 total 30.02 last 73.56) - open/close 2947 (time 1761315177.10 total 40.04 last 75.81) - open/close 3662 (time 1761315187.10 total 50.04 last 71.47) - open/close 4376 (time 1761315197.11 total 60.05 last 71.36) - open/close 5065 (time 1761315207.12 total 70.06 last 68.83) - open/close 5712 (time 1761315217.13 total 80.06 last 64.66) - open/close 6372 (time 1761315227.14 total 90.08 last 65.88) - open/close 7044 (time 1761315237.15 total 100.08 last 67.17) - open/close 7711 (time 1761315247.16 total 110.10 last 66.59) - open/close 8439 (time 1761315257.17 total 120.10 last 72.78) - open/close 9117 (time 1761315267.17 total 130.11 last 67.77) - open/close 9677 (time 1761315277.17 total 140.11 last 55.99) - open/close 10000 (time 1761315283.14 total 146.07 last 54.16) - open/close 10588 (time 1761315293.14 total 156.07 last 58.78) - open/close 11301 (time 1761315303.15 total 166.08 last 71.23) - open/close 12064 (time 1761315313.16 total 176.09 last 76.23) - open/close 12655 (time 1761315323.16 total 186.10 last 59.08) - open/close 13250 (time 1761315333.17 total 196.11 last 59.43) - open/close 13882 (time 1761315343.19 total 206.12 last 63.12) - open/close 14497 (time 1761315353.20 total 216.13 last 61.44) - open/close 15171 (time 1761315363.20 total 226.14 last 67.36) - open/close 15890 (time 1761315373.21 total 236.14 last 71.86) - open/close 16573 (time 1761315383.21 total 246.15 last 68.29) - open/close 17264 (time 1761315393.25 total 256.19 last 68.79) - open/close 17989 (time 1761315403.27 total 266.20 last 72.42) - open/close 18733 (time 1761315413.27 total 276.21 last 74.33) - open/close 19461 (time 1761315423.28 total 286.21 last 72.78) - open/close 20000 (time 1761315431.30 total 294.24 last 67.19) - open/close 20727 (time 1761315441.31 total 304.25 last 72.61) - open/close 21438 (time 1761315451.32 total 314.26 last 71.02) - open/close 22135 (time 1761315461.35 total 324.28 last 69.55) - open/close 22701 (time 1761315471.35 total 334.28 last 56.58) - open/close 23321 (time 1761315481.36 total 344.30 last 61.90) - open/close 24045 (time 1761315491.36 total 354.30 last 72.40) - open/close 24798 (time 1761315501.37 total 364.30 last 75.30) - open/close 25484 (time 1761315511.37 total 374.30 last 68.58) - open/close 26121 (time 1761315521.39 total 384.33 last 63.53) - open/close 26818 (time 1761315531.41 total 394.34 last 69.62) - open/close 27511 (time 1761315541.41 total 404.35 last 69.24) - open/close 28189 (time 1761315551.42 total 414.36 last 67.76) - open/close 28919 (time 1761315561.44 total 424.37 last 72.89) - open/close 29652 (time 1761315571.44 total 434.37 last 73.29) - open/close 30000 (time 1761315577.26 total 440.20 last 59.74) - open/close 30747 (time 1761315587.27 total 450.21 last 74.64) - open/close 31470 (time 1761315597.28 total 460.21 last 72.26) - open/close 32183 (time 1761315607.28 total 470.22 last 71.25) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 476.13 seconds: 68.46 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1183 (time 1761315627.01 total 10.00 last 118.29) - /unlink 2329 (time 1761315637.01 total 20.00 last 114.56) - /unlink 3531 (time 1761315647.02 total 30.01 last 120.16) - /unlink 4733 (time 1761315657.02 total 40.01 last 120.14) - /unlink 6009 (time 1761315667.03 total 50.02 last 127.56) - /unlink 7315 (time 1761315677.03 total 60.02 last 130.55) - /unlink 8475 (time 1761315687.03 total 70.02 last 115.95) - /unlink 9696 (time 1761315697.03 total 80.02 last 122.08) - /unlink 10000 (time 1761315699.55 total 82.54 last 120.73) - /unlink 11256 (time 1761315709.55 total 92.54 last 125.58) - /unlink 12466 (time 1761315719.56 total 102.55 last 120.96) - /unlink 13764 (time 1761315729.57 total 112.56 last 129.66) - /unlink 15004 (time 1761315739.57 total 122.56 last 123.97) - /unlink 16247 (time 1761315749.57 total 132.56 last 124.26) - /unlink 17495 (time 1761315759.57 total 142.56 last 124.79) - /unlink 18726 (time 1761315769.58 total 152.57 last 123.08) - /unlink 20000 (time 1761315779.39 total 162.38 last 129.86) - /unlink 21132 (time 1761315789.39 total 172.38 last 113.17) - /unlink 22368 (time 1761315799.39 total 182.38 last 123.58) - /unlink 23596 (time 1761315809.39 total 192.38 last 122.78) - /unlink 24785 (time 1761315819.40 total 202.39 last 118.82) - /unlink 26049 (time 1761315829.41 total 212.40 last 126.29) - /unlink 27254 (time 1761315839.41 total 222.40 last 120.48) - /unlink 28460 (time 1761315849.42 total 232.41 last 120.50) - /unlink 29751 (time 1761315859.43 total 242.42 last 128.97) - /unlink 30000 (time 1761315861.50 total 244.49 last 119.94) - /unlink 31237 (time 1761315871.51 total 254.50 last 123.69) - /unlink 32484 (time 1761315881.51 total 264.50 last 124.63) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 265.39 seconds: 122.82 ops/second Waiting for MDT destroys to complete - open/close 709 (time 1761315908.31 total 10.00 last 70.87) - open/close 1430 (time 1761315918.31 total 20.01 last 72.06) - open/close 2115 (time 1761315928.32 total 30.02 last 68.45) - open/close 2766 (time 1761315938.33 total 40.03 last 65.04) - open/close 3502 (time 1761315948.33 total 50.03 last 73.59) - open/close 4193 (time 1761315958.34 total 60.04 last 69.03) - open/close 4916 (time 1761315968.35 total 70.04 last 72.28) - open/close 5617 (time 1761315978.35 total 80.05 last 70.08) - open/close 6311 (time 1761315988.36 total 90.05 last 69.35) - open/close 6987 (time 1761315998.36 total 100.05 last 67.59) - open/close 7721 (time 1761316008.36 total 110.06 last 73.38) - open/close 8464 (time 1761316018.37 total 120.06 last 74.26) - open/close 9240 (time 1761316028.37 total 130.07 last 77.55) - open/close 9973 (time 1761316038.38 total 140.07 last 73.28) - open/close 10712 (time 1761316048.39 total 150.08 last 73.82) - open/close 11437 (time 1761316058.39 total 160.09 last 72.45) - open/close 12107 (time 1761316068.40 total 170.10 last 66.95) - open/close 12746 (time 1761316078.42 total 180.12 last 63.78) - open/close 13482 (time 1761316088.43 total 190.13 last 73.53) - open/close 14220 (time 1761316098.43 total 200.13 last 73.78) - open/close 14933 (time 1761316108.44 total 210.13 last 71.26) - open/close 15614 (time 1761316118.45 total 220.14 last 68.03) - open/close 16274 (time 1761316128.45 total 230.15 last 65.96) - open/close 16797 (time 1761316138.46 total 240.16 last 52.27) - open/close 17420 (time 1761316148.47 total 250.17 last 62.22) - open/close 18126 (time 1761316158.48 total 260.18 last 70.53) - open/close 18835 (time 1761316168.48 total 270.18 last 70.90) - open/close 19428 (time 1761316178.49 total 280.18 last 59.28) - open/close 20000 (time 1761316186.60 total 288.29 last 70.53) - open/close 20644 (time 1761316196.60 total 298.30 last 64.37) - open/close 21327 (time 1761316206.61 total 308.31 last 68.24) - open/close 22003 (time 1761316216.62 total 318.32 last 67.52) - open/close 22654 (time 1761316226.63 total 328.33 last 65.03) - open/close 23316 (time 1761316236.64 total 338.33 last 66.16) - open/close 23986 (time 1761316246.65 total 348.34 last 66.95) - open/close 24661 (time 1761316256.66 total 358.35 last 67.43) - open/close 25348 (time 1761316266.66 total 368.36 last 68.66) - open/close 26013 (time 1761316276.67 total 378.37 last 66.45) - open/close 26704 (time 1761316286.68 total 388.38 last 69.02) - open/close 27451 (time 1761316296.71 total 398.41 last 74.48) - open/close 28113 (time 1761316306.72 total 408.42 last 66.12) - open/close 28770 (time 1761316316.73 total 418.43 last 65.66) - open/close 29406 (time 1761316326.74 total 428.44 last 63.53) - open/close 30000 (time 1761316336.13 total 437.83 last 63.25) - open/close 30650 (time 1761316346.14 total 447.84 last 64.93) - open/close 31334 (time 1761316356.16 total 457.86 last 68.29) - open/close 32067 (time 1761316366.17 total 467.87 last 73.22) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 475.67 seconds: 68.52 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1040 (time 1761316388.96 total 10.01 last 103.92) - /unlink 2157 (time 1761316398.98 total 20.03 last 111.49) - /unlink 3244 (time 1761316408.98 total 30.03 last 108.63) - /unlink 4361 (time 1761316418.98 total 40.04 last 111.67) - /unlink 5533 (time 1761316428.99 total 50.04 last 117.14) - /unlink 6701 (time 1761316439.00 total 60.05 last 116.72) - /unlink 7897 (time 1761316449.00 total 70.05 last 119.57) - /unlink 9063 (time 1761316459.00 total 80.05 last 116.60) - /unlink 10000 (time 1761316467.31 total 88.36 last 112.80) - /unlink 11207 (time 1761316477.31 total 98.36 last 120.69) - /unlink 12503 (time 1761316487.31 total 108.36 last 129.60) - /unlink 13796 (time 1761316497.31 total 118.36 last 129.29) - /unlink 15052 (time 1761316507.31 total 128.36 last 125.55) - /unlink 16276 (time 1761316517.31 total 138.36 last 122.39) - /unlink 17530 (time 1761316527.31 total 148.37 last 125.39) - /unlink 18691 (time 1761316537.32 total 158.37 last 116.08) - /unlink 19924 (time 1761316547.32 total 168.37 last 123.24) - /unlink 21136 (time 1761316557.33 total 178.38 last 121.12) - /unlink 22311 (time 1761316567.33 total 188.38 last 117.50) - /unlink 23602 (time 1761316577.33 total 198.38 last 129.04) - /unlink 24898 (time 1761316587.33 total 208.38 last 129.58) - /unlink 26234 (time 1761316597.34 total 218.39 last 133.57) - /unlink 27544 (time 1761316607.34 total 228.39 last 130.92) - /unlink 28806 (time 1761316617.34 total 238.39 last 126.19) - /unlink 30000 (time 1761316626.47 total 247.52 last 130.85) - /unlink 31306 (time 1761316636.47 total 257.52 last 130.56) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 266.14 seconds: 122.47 ops/second Waiting for MDT destroys to complete - open/close 627 (time 1761316667.74 total 10.02 last 62.60) - open/close 1290 (time 1761316677.76 total 20.03 last 66.20) - open/close 2010 (time 1761316687.77 total 30.05 last 71.89) - open/close 2624 (time 1761316697.78 total 40.06 last 61.34) - open/close 3287 (time 1761316707.79 total 50.07 last 66.23) - open/close 4008 (time 1761316717.79 total 60.07 last 72.08) - open/close 4753 (time 1761316727.80 total 70.07 last 74.47) - open/close 5339 (time 1761316737.81 total 80.09 last 58.52) - open/close 6042 (time 1761316747.82 total 90.09 last 70.26) - open/close 6822 (time 1761316757.83 total 100.10 last 77.93) - open/close 7608 (time 1761316767.84 total 110.11 last 78.51) - open/close 8383 (time 1761316777.86 total 120.14 last 77.32) - open/close 9136 (time 1761316787.87 total 130.15 last 75.20) - open/close 9952 (time 1761316797.89 total 140.17 last 81.44) - open/close 10700 (time 1761316807.90 total 150.18 last 74.74) - open/close 11357 (time 1761316817.91 total 160.19 last 65.64) - open/close 12063 (time 1761316827.92 total 170.19 last 70.55) - open/close 12750 (time 1761316837.92 total 180.20 last 68.67) - open/close 13397 (time 1761316847.93 total 190.21 last 64.64) - open/close 14037 (time 1761316857.93 total 200.21 last 63.99) - open/close 14640 (time 1761316867.96 total 210.23 last 60.17) - open/close 15279 (time 1761316877.97 total 220.24 last 63.83) - open/close 15989 (time 1761316887.97 total 230.25 last 70.95) - open/close 16607 (time 1761316897.98 total 240.26 last 61.75) - open/close 17207 (time 1761316907.98 total 250.26 last 59.99) - open/close 17786 (time 1761316918.00 total 260.27 last 57.81) - open/close 18507 (time 1761316928.00 total 270.28 last 72.08) - open/close 19255 (time 1761316938.01 total 280.28 last 74.76) - open/close 19918 (time 1761316948.01 total 290.29 last 66.26) - open/close 20636 (time 1761316958.02 total 300.30 last 71.76) - open/close 21173 (time 1761316968.03 total 310.31 last 53.65) - open/close 21713 (time 1761316978.03 total 320.31 last 53.99) - open/close 22258 (time 1761316988.04 total 330.32 last 54.44) - open/close 22767 (time 1761316998.05 total 340.33 last 50.86) - open/close 23408 (time 1761317008.07 total 350.34 last 64.00) - open/close 24092 (time 1761317018.07 total 360.34 last 68.37) - open/close 24792 (time 1761317028.07 total 370.35 last 69.99) - open/close 25457 (time 1761317038.07 total 380.35 last 66.48) - open/close 26075 (time 1761317048.08 total 390.36 last 61.75) - open/close 26721 (time 1761317058.09 total 400.37 last 64.53) - open/close 27446 (time 1761317068.11 total 410.39 last 72.37) - open/close 28159 (time 1761317078.12 total 420.40 last 71.23) - open/close 28881 (time 1761317088.13 total 430.41 last 72.12) - open/close 29614 (time 1761317098.14 total 440.41 last 73.27) - open/close 30000 (time 1761317103.73 total 446.01 last 68.99) - open/close 30746 (time 1761317113.74 total 456.02 last 74.55) - open/close 31441 (time 1761317123.74 total 466.02 last 69.49) - open/close 32183 (time 1761317133.74 total 476.02 last 74.19) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 481.20 seconds: 67.74 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1442 (time 1761317152.40 total 10.00 last 144.16) - /unlink 2825 (time 1761317162.41 total 20.00 last 138.28) - /unlink 4029 (time 1761317172.42 total 30.02 last 120.27) - /unlink 5233 (time 1761317182.42 total 40.02 last 120.37) - /unlink 6486 (time 1761317192.42 total 50.02 last 125.24) - /unlink 7678 (time 1761317202.43 total 60.02 last 119.19) - /unlink 8857 (time 1761317212.43 total 70.02 last 117.89) - /unlink 10000 (time 1761317221.77 total 79.37 last 122.31) - /unlink 11242 (time 1761317231.77 total 89.37 last 124.16) - /unlink 12502 (time 1761317241.78 total 99.37 last 125.97) - /unlink 13729 (time 1761317251.78 total 109.38 last 122.60) - /unlink 14990 (time 1761317261.79 total 119.39 last 126.07) - /unlink 16327 (time 1761317271.79 total 129.39 last 133.64) - /unlink 17623 (time 1761317281.79 total 139.39 last 129.57) - /unlink 18852 (time 1761317291.80 total 149.39 last 122.87) - /unlink 20000 (time 1761317300.17 total 157.77 last 137.07) - /unlink 21458 (time 1761317310.17 total 167.77 last 145.76) - /unlink 22746 (time 1761317320.18 total 177.78 last 128.74) - /unlink 24006 (time 1761317330.19 total 187.78 last 125.92) - /unlink 25308 (time 1761317340.19 total 197.79 last 130.13) - /unlink 26393 (time 1761317350.19 total 207.79 last 108.46) - /unlink 27519 (time 1761317360.20 total 217.79 last 112.59) - /unlink 28676 (time 1761317370.21 total 227.81 last 115.56) - /unlink 29898 (time 1761317380.21 total 237.81 last 122.11) - /unlink 31127 (time 1761317390.22 total 247.81 last 122.88) - /unlink 32443 (time 1761317400.22 total 257.82 last 131.55) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 259.03 seconds: 125.83 ops/second Waiting for MDT destroys to complete - open/close 693 (time 1761317425.75 total 10.01 last 69.24) - open/close 1415 (time 1761317435.75 total 20.01 last 72.17) - open/close 2178 (time 1761317445.75 total 30.02 last 76.27) - open/close 2923 (time 1761317455.77 total 40.03 last 74.41) - open/close 3618 (time 1761317465.77 total 50.03 last 69.50) - open/close 4429 (time 1761317475.79 total 60.05 last 80.95) - open/close 5269 (time 1761317485.79 total 70.05 last 84.00) - open/close 6090 (time 1761317495.80 total 80.06 last 81.99) - open/close 6865 (time 1761317505.81 total 90.07 last 77.41) - open/close 7636 (time 1761317515.81 total 100.07 last 77.08) - open/close 8233 (time 1761317525.81 total 110.07 last 59.69) - open/close 8926 (time 1761317535.82 total 120.08 last 69.25) - open/close 9681 (time 1761317545.83 total 130.09 last 75.43) - open/close 10000 (time 1761317550.11 total 134.37 last 74.55) - open/close 10754 (time 1761317560.12 total 144.38 last 75.31) - open/close 11510 (time 1761317570.13 total 154.39 last 75.55) - open/close 12271 (time 1761317580.14 total 164.40 last 76.03) - open/close 13070 (time 1761317590.15 total 174.41 last 79.81) - open/close 13820 (time 1761317600.16 total 184.42 last 74.92) - open/close 14596 (time 1761317610.17 total 194.43 last 77.55) - open/close 15317 (time 1761317620.17 total 204.43 last 72.09) - open/close 15986 (time 1761317630.18 total 214.44 last 66.81) - open/close 16647 (time 1761317640.19 total 224.45 last 66.01) - open/close 17401 (time 1761317650.19 total 234.45 last 75.40) - open/close 18089 (time 1761317660.21 total 244.47 last 68.71) - open/close 18780 (time 1761317670.21 total 254.47 last 69.06) - open/close 19489 (time 1761317680.22 total 264.48 last 70.87) - open/close 20000 (time 1761317687.30 total 271.56 last 72.13) - open/close 20735 (time 1761317697.31 total 281.57 last 73.45) - open/close 21432 (time 1761317707.32 total 291.58 last 69.65) - open/close 22189 (time 1761317717.32 total 301.58 last 75.70) - open/close 22888 (time 1761317727.32 total 311.58 last 69.89) - open/close 23660 (time 1761317737.32 total 321.58 last 77.16) - open/close 24424 (time 1761317747.32 total 331.58 last 76.39) - open/close 25151 (time 1761317757.33 total 341.59 last 72.67) - open/close 25881 (time 1761317767.34 total 351.60 last 72.91) - open/close 26638 (time 1761317777.34 total 361.60 last 75.69) - open/close 27407 (time 1761317787.36 total 371.62 last 76.79) - open/close 28087 (time 1761317797.36 total 381.62 last 67.97) - open/close 28782 (time 1761317807.37 total 391.63 last 69.45) - open/close 29520 (time 1761317817.38 total 401.64 last 73.71) - open/close 30000 (time 1761317823.77 total 408.03 last 75.08) - open/close 30669 (time 1761317833.78 total 418.04 last 66.88) - open/close 31462 (time 1761317843.78 total 428.04 last 79.24) - open/close 32233 (time 1761317853.79 total 438.05 last 77.07) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 442.69 seconds: 73.63 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1263 (time 1761317871.60 total 10.00 last 126.28) - /unlink 2474 (time 1761317881.62 total 20.02 last 120.87) - /unlink 3745 (time 1761317891.62 total 30.02 last 127.09) - /unlink 5090 (time 1761317901.62 total 40.02 last 134.47) - /unlink 6445 (time 1761317911.62 total 50.03 last 135.45) - /unlink 7888 (time 1761317921.63 total 60.03 last 144.23) - /unlink 9586 (time 1761317931.63 total 70.04 last 169.71) - /unlink 10000 (time 1761317933.83 total 72.23 last 188.54) - /unlink 12042 (time 1761317943.83 total 82.24 last 204.12) - /unlink 13663 (time 1761317953.83 total 92.24 last 162.05) - /unlink 15001 (time 1761317963.84 total 102.25 last 133.73) - /unlink 16748 (time 1761317973.84 total 112.25 last 174.68) - /unlink 18446 (time 1761317983.84 total 122.25 last 169.76) - /unlink 19948 (time 1761317993.85 total 132.25 last 150.12) - /unlink 21431 (time 1761318003.85 total 142.26 last 148.28) - /unlink 22929 (time 1761318013.85 total 152.26 last 149.74) - /unlink 24429 (time 1761318023.86 total 162.26 last 149.91) - /unlink 25817 (time 1761318033.86 total 172.27 last 138.78) - /unlink 27181 (time 1761318043.86 total 182.27 last 136.40) - /unlink 28659 (time 1761318053.87 total 192.27 last 147.72) - /unlink 29981 (time 1761318063.87 total 202.28 last 132.07) - /unlink 31257 (time 1761318073.88 total 212.28 last 127.57) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 221.97 seconds: 146.84 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing unload_modules_local oleg216-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg216-server: LNET ready to unload oleg216-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 PASS 106 (3879s) Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 oleg216-server: oleg216-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: detected 4 online CPUs by sysfs oleg216-server: Force libcfs to create 2 CPU partitions oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg216-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg216-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4125 sec ========== 11:03:12 (1761318192)