-----============= acceptance-small: conf-sanity ============----- Fri Dec 5 07:15:31 EST 2025 oleg242-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg242-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg242-server oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg242-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg242-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg242-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg242-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:18:42 (1764937122) Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg242-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg242-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg242-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock oleg242-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 678 (time 1764937241.12 total 10.00 last 67.79) - open/close 1339 (time 1764937251.12 total 20.00 last 66.08) - open/close 1985 (time 1764937261.14 total 30.03 last 64.46) - open/close 2643 (time 1764937271.16 total 40.04 last 65.68) - open/close 3350 (time 1764937281.17 total 50.06 last 70.59) - open/close 4010 (time 1764937291.18 total 60.07 last 65.96) - open/close 4677 (time 1764937301.20 total 70.08 last 66.57) - open/close 5353 (time 1764937311.21 total 80.10 last 67.53) - open/close 6086 (time 1764937321.21 total 90.10 last 73.27) - open/close 6794 (time 1764937331.22 total 100.10 last 70.77) - open/close 7536 (time 1764937341.23 total 110.11 last 74.14) - open/close 8262 (time 1764937351.23 total 120.12 last 72.55) - open/close 8953 (time 1764937361.24 total 130.12 last 69.07) - open/close 9640 (time 1764937371.25 total 140.13 last 68.65) - open/close 10000 (time 1764937377.51 total 146.40 last 57.46) - open/close 10602 (time 1764937387.53 total 156.42 last 60.08) - open/close 11293 (time 1764937397.54 total 166.42 last 69.06) - open/close 11997 (time 1764937407.56 total 176.45 last 70.25) - open/close 12705 (time 1764937417.58 total 186.47 last 70.65) - open/close 13401 (time 1764937427.58 total 196.47 last 69.59) - open/close 14061 (time 1764937437.60 total 206.48 last 65.90) - open/close 14652 (time 1764937447.61 total 216.49 last 59.04) - open/close 15407 (time 1764937457.62 total 226.51 last 75.41) - open/close 16115 (time 1764937467.63 total 236.52 last 70.71) - open/close 16797 (time 1764937477.63 total 246.52 last 68.19) - open/close 17511 (time 1764937487.64 total 256.53 last 71.35) - open/close 18216 (time 1764937497.64 total 266.53 last 70.50) - open/close 18901 (time 1764937507.64 total 276.53 last 68.49) - open/close 19604 (time 1764937517.65 total 286.54 last 70.24) - open/close 20000 (time 1764937523.17 total 292.05 last 71.79) - open/close 20650 (time 1764937533.17 total 302.06 last 64.96) - open/close 21333 (time 1764937543.17 total 312.06 last 68.29) - open/close 22060 (time 1764937553.18 total 322.07 last 72.65) - open/close 22767 (time 1764937563.19 total 332.08 last 70.61) - open/close 23430 (time 1764937573.19 total 342.08 last 66.29) - open/close 24160 (time 1764937583.20 total 352.09 last 72.94) - open/close 24855 (time 1764937593.20 total 362.09 last 69.50) - open/close 25483 (time 1764937603.21 total 372.10 last 62.73) - open/close 26191 (time 1764937613.23 total 382.11 last 70.70) - open/close 26889 (time 1764937623.23 total 392.12 last 69.77) - open/close 27542 (time 1764937633.24 total 402.13 last 65.24) - open/close 28265 (time 1764937643.25 total 412.13 last 72.26) - open/close 28927 (time 1764937653.26 total 422.15 last 66.10) - open/close 29632 (time 1764937663.27 total 432.16 last 70.42) - open/close 30000 (time 1764937668.28 total 437.17 last 73.43) - open/close 30697 (time 1764937678.29 total 447.17 last 69.68) - open/close 31394 (time 1764937688.29 total 457.17 last 69.70) - open/close 32110 (time 1764937698.29 total 467.18 last 71.58) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 473.57 seconds: 68.83 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1320 (time 1764937718.47 total 10.01 last 131.92) - /unlink 2452 (time 1764937728.47 total 20.01 last 113.18) - /unlink 3633 (time 1764937738.47 total 30.01 last 118.09) - /unlink 4829 (time 1764937748.47 total 40.01 last 119.58) - /unlink 5973 (time 1764937758.48 total 50.02 last 114.30) - /unlink 7198 (time 1764937768.48 total 60.02 last 122.50) - /unlink 8448 (time 1764937778.49 total 70.02 last 124.98) - /unlink 9671 (time 1764937788.49 total 80.02 last 122.29) - /unlink 10000 (time 1764937791.19 total 82.73 last 121.69) - /unlink 11306 (time 1764937801.19 total 92.73 last 130.59) - /unlink 12636 (time 1764937811.20 total 102.73 last 132.91) - /unlink 13872 (time 1764937821.20 total 112.74 last 123.55) - /unlink 15129 (time 1764937831.20 total 122.74 last 125.69) - /unlink 16309 (time 1764937841.20 total 132.74 last 117.98) - /unlink 17499 (time 1764937851.21 total 142.74 last 118.95) - /unlink 18685 (time 1764937861.22 total 152.75 last 118.50) - /unlink 19903 (time 1764937871.22 total 162.75 last 121.80) - /unlink 21244 (time 1764937881.22 total 172.76 last 134.05) - /unlink 22416 (time 1764937891.22 total 182.76 last 117.20) - /unlink 23429 (time 1764937901.23 total 192.76 last 101.21) - /unlink 24597 (time 1764937911.23 total 202.76 last 116.80) - /unlink 25810 (time 1764937921.24 total 212.77 last 121.22) - /unlink 26997 (time 1764937931.25 total 222.78 last 118.57) - /unlink 28286 (time 1764937941.26 total 232.79 last 128.79) - /unlink 29547 (time 1764937951.26 total 242.79 last 126.08) - /unlink 30000 (time 1764937954.73 total 246.27 last 130.39) - /unlink 31323 (time 1764937964.74 total 256.27 last 132.23) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 266.14 seconds: 122.47 ops/second Waiting for MDT destroys to complete - open/close 800 (time 1764937995.67 total 10.01 last 79.90) - open/close 1583 (time 1764938005.67 total 20.01 last 78.29) - open/close 2392 (time 1764938015.68 total 30.02 last 80.84) - open/close 3149 (time 1764938025.68 total 40.03 last 75.64) - open/close 3886 (time 1764938035.70 total 50.05 last 73.56) - open/close 4668 (time 1764938045.71 total 60.05 last 78.15) - open/close 5399 (time 1764938055.72 total 70.06 last 73.06) - open/close 6105 (time 1764938065.72 total 80.07 last 70.54) - open/close 6814 (time 1764938075.73 total 90.07 last 70.89) - open/close 7546 (time 1764938085.74 total 100.08 last 73.11) - open/close 8330 (time 1764938095.75 total 110.09 last 78.34) - open/close 9045 (time 1764938105.75 total 120.09 last 71.50) - open/close 9800 (time 1764938115.75 total 130.09 last 75.50) - open/close 10000 (time 1764938118.29 total 132.64 last 78.60) - open/close 10767 (time 1764938128.31 total 142.66 last 76.55) - open/close 11512 (time 1764938138.31 total 152.66 last 74.50) - open/close 12237 (time 1764938148.33 total 162.67 last 72.37) - open/close 12933 (time 1764938158.34 total 172.68 last 69.56) - open/close 13666 (time 1764938168.34 total 182.69 last 73.24) - open/close 14373 (time 1764938178.35 total 192.69 last 70.69) - open/close 15112 (time 1764938188.35 total 202.69 last 73.86) - open/close 15874 (time 1764938198.35 total 212.70 last 76.18) - open/close 16626 (time 1764938208.36 total 222.71 last 75.12) - open/close 17498 (time 1764938218.37 total 232.71 last 87.16) - open/close 18446 (time 1764938228.37 total 242.72 last 94.77) - open/close 19198 (time 1764938238.40 total 252.74 last 75.02) - open/close 19920 (time 1764938248.41 total 262.76 last 72.10) - open/close 20554 (time 1764938258.41 total 272.76 last 63.38) - open/close 21264 (time 1764938268.43 total 282.77 last 70.89) - open/close 21986 (time 1764938278.44 total 292.78 last 72.13) - open/close 22670 (time 1764938288.45 total 302.79 last 68.33) - open/close 23415 (time 1764938298.45 total 312.80 last 74.47) - open/close 24093 (time 1764938308.47 total 322.81 last 67.73) - open/close 24791 (time 1764938318.47 total 332.81 last 69.78) - open/close 25593 (time 1764938328.48 total 342.82 last 80.11) - open/close 26375 (time 1764938338.49 total 352.83 last 78.13) - open/close 27129 (time 1764938348.49 total 362.83 last 75.40) - open/close 27896 (time 1764938358.50 total 372.84 last 76.62) - open/close 28675 (time 1764938368.51 total 382.86 last 77.80) - open/close 29403 (time 1764938378.52 total 392.86 last 72.77) - open/close 30000 (time 1764938386.36 total 400.71 last 76.09) - open/close 30743 (time 1764938396.38 total 410.72 last 74.19) - open/close 31458 (time 1764938406.38 total 420.72 last 71.49) - open/close 32193 (time 1764938416.38 total 430.73 last 73.47) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 436.68 seconds: 74.64 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1177 (time 1764938436.69 total 10.00 last 117.69) - /unlink 2458 (time 1764938446.70 total 20.01 last 128.03) - /unlink 3741 (time 1764938456.70 total 30.01 last 128.24) - /unlink 4938 (time 1764938466.71 total 40.01 last 119.67) - /unlink 6119 (time 1764938476.72 total 50.03 last 117.94) - /unlink 7403 (time 1764938486.73 total 60.03 last 128.33) - /unlink 8595 (time 1764938496.73 total 70.04 last 119.16) - /unlink 9748 (time 1764938506.73 total 80.04 last 115.24) - /unlink 10000 (time 1764938509.21 total 82.52 last 101.65) - /unlink 11210 (time 1764938519.23 total 92.53 last 120.86) - /unlink 12196 (time 1764938529.23 total 102.54 last 98.53) - /unlink 13257 (time 1764938539.24 total 112.54 last 106.06) - /unlink 14498 (time 1764938549.24 total 122.55 last 124.05) - /unlink 15771 (time 1764938559.24 total 132.55 last 127.25) - /unlink 17038 (time 1764938569.26 total 142.56 last 126.56) - /unlink 18331 (time 1764938579.26 total 152.57 last 129.23) - /unlink 19578 (time 1764938589.27 total 162.57 last 124.64) - /unlink 20000 (time 1764938592.27 total 165.57 last 140.67) - /unlink 21253 (time 1764938602.27 total 175.58 last 125.23) - /unlink 22483 (time 1764938612.28 total 185.58 last 122.94) - /unlink 23735 (time 1764938622.28 total 195.59 last 125.10) - /unlink 24917 (time 1764938632.29 total 205.60 last 118.14) - /unlink 26174 (time 1764938642.30 total 215.61 last 125.52) - /unlink 27411 (time 1764938652.31 total 225.62 last 123.63) - /unlink 28631 (time 1764938662.31 total 235.62 last 121.99) - /unlink 29912 (time 1764938672.32 total 245.62 last 128.01) - /unlink 31135 (time 1764938682.32 total 255.63 last 122.23) - /unlink 32432 (time 1764938692.33 total 265.63 last 129.64) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 267.12 seconds: 122.02 ops/second Waiting for MDT destroys to complete - open/close 726 (time 1764938718.94 total 10.01 last 72.52) - open/close 1396 (time 1764938728.97 total 20.04 last 66.84) - open/close 2149 (time 1764938738.97 total 30.04 last 75.27) - open/close 2898 (time 1764938748.99 total 40.06 last 74.77) - open/close 3628 (time 1764938758.99 total 50.06 last 72.96) - open/close 4381 (time 1764938769.00 total 60.07 last 75.24) - open/close 5174 (time 1764938779.00 total 70.07 last 79.28) - open/close 5947 (time 1764938789.01 total 80.08 last 77.24) - open/close 6744 (time 1764938799.02 total 90.09 last 79.65) - open/close 7531 (time 1764938809.02 total 100.09 last 78.64) - open/close 8288 (time 1764938819.03 total 110.10 last 75.63) - open/close 9054 (time 1764938829.04 total 120.11 last 76.56) - open/close 9826 (time 1764938839.06 total 130.13 last 77.04) - open/close 10000 (time 1764938841.36 total 132.43 last 75.68) - open/close 10792 (time 1764938851.37 total 142.44 last 79.12) - open/close 11654 (time 1764938861.38 total 152.44 last 86.16) - open/close 12444 (time 1764938871.38 total 162.45 last 78.94) - open/close 13187 (time 1764938881.39 total 172.46 last 74.22) - open/close 13943 (time 1764938891.40 total 182.47 last 75.58) - open/close 14785 (time 1764938901.40 total 192.47 last 84.17) - open/close 15600 (time 1764938911.41 total 202.48 last 81.42) - open/close 16469 (time 1764938921.41 total 212.48 last 86.88) - open/close 17304 (time 1764938931.41 total 222.48 last 83.49) - open/close 18145 (time 1764938941.42 total 232.49 last 84.07) - open/close 19044 (time 1764938951.42 total 242.49 last 89.90) - open/close 19853 (time 1764938961.42 total 252.49 last 80.87) - open/close 20621 (time 1764938971.43 total 262.50 last 76.71) - open/close 21338 (time 1764938981.44 total 272.50 last 71.69) - open/close 22040 (time 1764938991.45 total 282.52 last 70.09) - open/close 22796 (time 1764939001.45 total 292.52 last 75.60) - open/close 23454 (time 1764939011.45 total 302.52 last 65.77) - open/close 24147 (time 1764939021.47 total 312.53 last 69.22) - open/close 24847 (time 1764939031.48 total 322.55 last 69.91) - open/close 25596 (time 1764939041.48 total 332.55 last 74.89) - open/close 26297 (time 1764939051.48 total 342.55 last 70.07) - open/close 27044 (time 1764939061.49 total 352.56 last 74.65) - open/close 27745 (time 1764939071.51 total 362.57 last 70.00) - open/close 28445 (time 1764939081.51 total 372.58 last 70.00) - open/close 29142 (time 1764939091.52 total 382.58 last 69.63) - open/close 29853 (time 1764939101.52 total 392.59 last 71.09) - open/close 30000 (time 1764939103.53 total 394.60 last 72.93) - open/close 30720 (time 1764939113.54 total 404.61 last 71.93) - open/close 31438 (time 1764939123.54 total 414.61 last 71.79) - open/close 32138 (time 1764939133.55 total 424.62 last 69.96) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 430.88 seconds: 75.65 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1361 (time 1764939153.16 total 10.00 last 136.04) - /unlink 2577 (time 1764939163.17 total 20.01 last 121.49) - /unlink 3767 (time 1764939173.18 total 30.02 last 118.95) - /unlink 4945 (time 1764939183.18 total 40.02 last 117.76) - /unlink 6145 (time 1764939193.18 total 50.03 last 119.93) - /unlink 7326 (time 1764939203.19 total 60.03 last 118.06) - /unlink 8604 (time 1764939213.19 total 70.03 last 127.78) - /unlink 10000 (time 1764939222.85 total 79.69 last 144.52) - /unlink 11227 (time 1764939232.85 total 89.69 last 122.69) - /unlink 12376 (time 1764939242.86 total 99.70 last 114.83) - /unlink 13535 (time 1764939252.87 total 109.71 last 115.79) - /unlink 14872 (time 1764939262.87 total 119.71 last 133.60) - /unlink 16116 (time 1764939272.88 total 129.72 last 124.33) - /unlink 17370 (time 1764939282.89 total 139.74 last 125.20) - /unlink 18619 (time 1764939292.90 total 149.74 last 124.85) - /unlink 19943 (time 1764939302.90 total 159.74 last 132.35) - /unlink 21230 (time 1764939312.91 total 169.75 last 128.65) - /unlink 22515 (time 1764939322.91 total 179.75 last 128.44) - /unlink 23672 (time 1764939332.92 total 189.76 last 115.59) - /unlink 25002 (time 1764939342.92 total 199.76 last 132.99) - /unlink 26376 (time 1764939352.93 total 209.77 last 137.29) - /unlink 27855 (time 1764939362.93 total 219.77 last 147.89) - /unlink 29108 (time 1764939372.93 total 229.77 last 125.30) - /unlink 30000 (time 1764939379.88 total 236.72 last 128.37) - /unlink 31260 (time 1764939389.88 total 246.72 last 125.98) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 256.28 seconds: 127.18 ops/second Waiting for MDT destroys to complete - open/close 717 (time 1764939424.64 total 10.01 last 71.66) - open/close 1443 (time 1764939434.64 total 20.01 last 72.55) - open/close 2129 (time 1764939444.65 total 30.02 last 68.55) - open/close 2978 (time 1764939454.66 total 40.03 last 84.86) - open/close 3713 (time 1764939464.67 total 50.04 last 73.40) - open/close 4373 (time 1764939474.67 total 60.04 last 65.96) - open/close 5091 (time 1764939484.68 total 70.05 last 71.77) - open/close 5849 (time 1764939494.68 total 80.05 last 75.77) - open/close 6561 (time 1764939504.69 total 90.06 last 71.17) - open/close 7217 (time 1764939514.70 total 100.07 last 65.54) - open/close 7896 (time 1764939524.70 total 110.07 last 67.86) - open/close 8615 (time 1764939534.72 total 120.09 last 71.79) - open/close 9360 (time 1764939544.73 total 130.10 last 74.42) - open/close 10000 (time 1764939553.89 total 139.27 last 69.83) - open/close 10771 (time 1764939563.90 total 149.27 last 77.07) - open/close 11501 (time 1764939573.90 total 159.27 last 72.98) - open/close 12163 (time 1764939583.91 total 169.28 last 66.14) - open/close 12878 (time 1764939593.91 total 179.28 last 71.47) - open/close 13633 (time 1764939603.92 total 189.29 last 75.46) - open/close 14458 (time 1764939613.93 total 199.30 last 82.44) - open/close 15303 (time 1764939623.93 total 209.30 last 84.45) - open/close 16130 (time 1764939633.94 total 219.31 last 82.66) - open/close 16937 (time 1764939643.95 total 229.32 last 80.62) - open/close 17718 (time 1764939653.96 total 239.33 last 78.03) - open/close 18529 (time 1764939663.97 total 249.34 last 81.01) - open/close 19320 (time 1764939673.97 total 259.34 last 79.04) - open/close 20000 (time 1764939681.99 total 267.36 last 84.80) - open/close 20877 (time 1764939692.00 total 277.37 last 87.63) - open/close 21611 (time 1764939702.00 total 287.37 last 73.39) - open/close 22439 (time 1764939712.00 total 297.37 last 82.79) - open/close 23231 (time 1764939722.01 total 307.39 last 79.11) - open/close 24098 (time 1764939732.02 total 317.39 last 86.65) - open/close 25006 (time 1764939742.02 total 327.39 last 90.78) - open/close 25806 (time 1764939752.03 total 337.40 last 79.97) - open/close 26759 (time 1764939762.04 total 347.41 last 95.20) - open/close 27670 (time 1764939772.04 total 357.41 last 91.04) - open/close 28536 (time 1764939782.05 total 367.42 last 86.55) - open/close 29345 (time 1764939792.06 total 377.43 last 80.80) - open/close 30000 (time 1764939800.55 total 385.92 last 77.19) - open/close 30789 (time 1764939810.56 total 395.93 last 78.80) - open/close 31615 (time 1764939820.56 total 405.93 last 82.57) - open/close 32488 (time 1764939830.57 total 415.94 last 87.29) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 416.89 seconds: 78.18 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1624 (time 1764939843.81 total 10.00 last 162.33) - /unlink 3375 (time 1764939853.82 total 20.01 last 175.03) - /unlink 5232 (time 1764939863.82 total 30.01 last 185.63) - /unlink 7188 (time 1764939873.83 total 40.02 last 195.49) - /unlink 8974 (time 1764939883.84 total 50.03 last 178.35) - /unlink 10000 (time 1764939891.12 total 57.31 last 140.93) - /unlink 11840 (time 1764939901.13 total 67.32 last 183.83) - /unlink 13939 (time 1764939911.13 total 77.32 last 209.84) - /unlink 15834 (time 1764939921.13 total 87.33 last 189.47) - /unlink 17572 (time 1764939931.14 total 97.33 last 173.77) - /unlink 19075 (time 1764939941.14 total 107.33 last 150.29) - /unlink 20000 (time 1764939946.34 total 112.54 last 177.64) - /unlink 21834 (time 1764939956.35 total 122.54 last 183.35) - /unlink 23536 (time 1764939966.35 total 132.54 last 170.14) - /unlink 25079 (time 1764939976.35 total 142.54 last 154.25) - /unlink 26828 (time 1764939986.35 total 152.55 last 174.87) - /unlink 28726 (time 1764939996.36 total 162.55 last 189.78) - /unlink 30000 (time 1764940004.36 total 170.55 last 159.18) - /unlink 31769 (time 1764940014.36 total 180.56 last 176.79) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 186.17 seconds: 175.08 ops/second Waiting for MDT destroys to complete - open/close 859 (time 1764940042.31 total 10.02 last 85.69) - open/close 1666 (time 1764940052.32 total 20.02 last 80.70) - open/close 2549 (time 1764940062.32 total 30.03 last 88.25) - open/close 3348 (time 1764940072.33 total 40.04 last 79.83) - open/close 4183 (time 1764940082.33 total 50.04 last 83.47) - open/close 4958 (time 1764940092.34 total 60.05 last 77.45) - open/close 5699 (time 1764940102.34 total 70.05 last 74.06) - open/close 6509 (time 1764940112.36 total 80.06 last 80.92) - open/close 7471 (time 1764940122.36 total 90.07 last 96.15) - open/close 8576 (time 1764940132.37 total 100.08 last 110.43) - open/close 9778 (time 1764940142.37 total 110.08 last 120.15) - open/close 10811 (time 1764940152.38 total 120.08 last 103.25) - open/close 11942 (time 1764940162.38 total 130.09 last 113.07) - open/close 13135 (time 1764940172.38 total 140.09 last 119.22) - open/close 14422 (time 1764940182.39 total 150.10 last 128.68) - open/close 15563 (time 1764940192.40 total 160.11 last 113.97) - open/close 16928 (time 1764940202.40 total 170.11 last 136.43) - open/close 18572 (time 1764940212.40 total 180.11 last 164.36) - open/close 19629 (time 1764940222.42 total 190.13 last 105.55) - open/close 20000 (time 1764940226.05 total 193.76 last 102.21) - open/close 21080 (time 1764940236.06 total 203.76 last 107.93) - open/close 22044 (time 1764940246.06 total 213.77 last 96.40) - open/close 23073 (time 1764940256.06 total 223.77 last 102.86) - open/close 24031 (time 1764940266.07 total 233.78 last 95.74) - open/close 25242 (time 1764940276.08 total 243.79 last 120.94) - open/close 26413 (time 1764940286.08 total 253.79 last 117.07) - open/close 27375 (time 1764940296.09 total 263.80 last 96.16) - open/close 28314 (time 1764940306.09 total 273.80 last 93.88) - open/close 29410 (time 1764940316.09 total 283.80 last 109.52) - open/close 30000 (time 1764940321.37 total 289.08 last 111.80) - open/close 31062 (time 1764940331.38 total 299.09 last 106.14) - open/close 32083 (time 1764940341.39 total 309.10 last 102.00) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 314.69 seconds: 103.58 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1889 (time 1764940359.37 total 10.00 last 188.86) - /unlink 3967 (time 1764940369.37 total 20.00 last 207.77) - /unlink 5761 (time 1764940379.37 total 30.01 last 179.29) - /unlink 7487 (time 1764940389.38 total 40.01 last 172.54) - /unlink 9809 (time 1764940399.38 total 50.02 last 232.14) - /unlink 11984 (time 1764940409.38 total 60.02 last 217.49) - /unlink 14200 (time 1764940419.38 total 70.02 last 221.54) - /unlink 16718 (time 1764940429.39 total 80.02 last 251.71) - /unlink 18685 (time 1764940439.39 total 90.02 last 196.67) - /unlink 20000 (time 1764940445.19 total 95.82 last 226.74) - /unlink 21681 (time 1764940455.19 total 105.82 last 168.09) - /unlink 23566 (time 1764940465.19 total 115.83 last 188.43) - /unlink 25651 (time 1764940475.19 total 125.83 last 208.47) - /unlink 27865 (time 1764940485.20 total 135.83 last 221.34) - /unlink 30000 (time 1764940494.73 total 145.37 last 223.81) - /unlink 32438 (time 1764940504.74 total 155.37 last 243.73) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 155.92 seconds: 209.04 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg242-server: LNET ready to unload oleg242-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 PASS 106 (3424s) Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3650 sec ========== 08:16:23 (1764940583)