-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 12:05:16 EDT 2025 oleg122-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg122-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server oleg122-server: oleg122-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg122-server: detected 4 online CPUs by sysfs oleg122-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg122-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg122-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg122-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg122-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg122-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:07:19 (1761322039) Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f) oleg122-server: oleg122-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg122-server: detected 4 online CPUs by sysfs oleg122-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg122-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg122-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg122-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock oleg122-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 2276 (time 1761322103.73 total 10.00 last 227.56) - open/close 4808 (time 1761322113.73 total 20.00 last 253.13) - open/close 7441 (time 1761322123.73 total 30.01 last 263.27) - open/close 9832 (time 1761322133.74 total 40.01 last 238.99) - open/close 12224 (time 1761322143.74 total 50.01 last 239.17) - open/close 14564 (time 1761322153.74 total 60.01 last 233.97) - open/close 16705 (time 1761322163.74 total 70.02 last 214.03) - open/close 18974 (time 1761322173.74 total 80.02 last 226.86) - open/close 20000 (time 1761322178.13 total 84.40 last 234.13) - open/close 22147 (time 1761322188.13 total 94.40 last 214.59) - open/close 24526 (time 1761322198.13 total 104.41 last 237.89) - open/close 27819 (time 1761322208.13 total 114.41 last 329.28) - open/close 30000 (time 1761322215.03 total 121.30 last 316.42) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 129.40 seconds: 251.88 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4559 (time 1761322234.64 total 10.00 last 455.80) - /unlink 8208 (time 1761322244.64 total 20.00 last 364.81) - /unlink 10000 (time 1761322249.16 total 24.52 last 396.63) - /unlink 14367 (time 1761322259.16 total 34.53 last 436.54) - /unlink 19021 (time 1761322269.17 total 44.53 last 465.38) - /unlink 20000 (time 1761322271.39 total 46.76 last 439.23) - /unlink 25522 (time 1761322281.40 total 56.76 last 552.12) - /unlink 29927 (time 1761322291.40 total 66.76 last 440.42) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 73.02 seconds: 446.39 ops/second Waiting for MDT destroys to complete - open/close 2772 (time 1761322320.58 total 10.00 last 277.11) - open/close 5158 (time 1761322330.58 total 20.01 last 238.56) - open/close 7800 (time 1761322340.58 total 30.01 last 264.19) - open/close 10000 (time 1761322349.94 total 39.36 last 235.06) - open/close 11950 (time 1761322359.94 total 49.37 last 194.94) - open/close 14445 (time 1761322369.94 total 59.37 last 249.43) - open/close 16740 (time 1761322379.95 total 69.37 last 229.41) - open/close 19460 (time 1761322389.95 total 79.38 last 271.94) - open/close 22047 (time 1761322399.95 total 89.38 last 258.69) - open/close 24702 (time 1761322409.95 total 99.38 last 265.43) - open/close 27643 (time 1761322419.95 total 109.38 last 294.07) - open/close 30000 (time 1761322428.17 total 117.60 last 286.73) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 127.35 seconds: 255.94 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4629 (time 1761322449.49 total 10.00 last 462.85) - /unlink 10000 (time 1761322459.22 total 19.74 last 551.70) - /unlink 16660 (time 1761322469.22 total 29.74 last 665.97) - /unlink 20000 (time 1761322474.30 total 34.82 last 657.44) - /unlink 26263 (time 1761322484.30 total 44.82 last 626.29) - /unlink 30000 (time 1761322490.11 total 50.62 last 643.51) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 54.84 seconds: 594.34 ops/second Waiting for MDT destroys to complete - open/close 3358 (time 1761322524.17 total 10.00 last 335.74) - open/close 6504 (time 1761322534.17 total 20.01 last 314.48) - open/close 9725 (time 1761322544.18 total 30.01 last 322.03) - open/close 12971 (time 1761322554.18 total 40.01 last 324.58) - open/close 16246 (time 1761322564.18 total 50.01 last 327.50) - open/close 19251 (time 1761322574.18 total 60.01 last 300.45) - open/close 20000 (time 1761322576.18 total 62.01 last 373.80) - open/close 23102 (time 1761322586.18 total 72.01 last 310.18) - open/close 26302 (time 1761322596.18 total 82.01 last 319.99) - open/close 28774 (time 1761322606.18 total 92.02 last 247.16) - open/close 30000 (time 1761322611.39 total 97.22 last 235.58) - open/close 32466 (time 1761322621.39 total 107.22 last 246.53) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 107.85 seconds: 302.22 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 5495 (time 1761322633.64 total 10.00 last 549.44) - /unlink 10000 (time 1761322640.73 total 17.09 last 635.14) - /unlink 16595 (time 1761322650.73 total 27.09 last 659.46) - /unlink 20000 (time 1761322655.34 total 31.71 last 738.46) - /unlink 25720 (time 1761322665.34 total 41.71 last 571.97) - /unlink 30000 (time 1761322672.84 total 49.21 last 570.47) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 53.60 seconds: 608.06 ops/second Waiting for MDT destroys to complete - open/close 2562 (time 1761322694.22 total 10.00 last 256.12) - open/close 5693 (time 1761322704.22 total 20.00 last 313.06) - open/close 8878 (time 1761322714.22 total 30.01 last 318.41) - open/close 10000 (time 1761322717.50 total 33.28 last 342.65) - open/close 13537 (time 1761322727.50 total 43.28 last 353.67) - open/close 17616 (time 1761322737.50 total 53.28 last 407.87) - open/close 20000 (time 1761322744.43 total 60.21 last 344.04) - open/close 24220 (time 1761322754.43 total 70.21 last 421.98) - open/close 28700 (time 1761322764.43 total 80.21 last 447.96) - open/close 30000 (time 1761322768.08 total 83.86 last 356.32) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 91.31 seconds: 356.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8404 (time 1761322786.94 total 10.00 last 840.32) - /unlink 16798 (time 1761322796.94 total 20.00 last 839.40) - /unlink 20000 (time 1761322801.94 total 25.00 last 640.19) - /unlink 21857 (time 1761322811.94 total 35.01 last 185.65) - /unlink 23507 (time 1761322821.95 total 45.01 last 164.92) - /unlink 25073 (time 1761322831.95 total 55.01 last 156.57) - /unlink 26421 (time 1761322841.96 total 65.02 last 134.63) - /unlink 27614 (time 1761322852.04 total 75.11 last 118.33) - /unlink 28626 (time 1761322862.05 total 85.11 last 101.13) - /unlink 30000 (time 1761322872.04 total 95.10 last 137.59) - /unlink 31595 (time 1761322882.04 total 105.10 last 159.42) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 111.10 seconds: 293.39 ops/second Waiting for MDT destroys to complete - open/close 1116 (time 1761322908.23 total 10.01 last 111.45) - open/close 2128 (time 1761322918.23 total 20.02 last 101.16) - open/close 3112 (time 1761322928.24 total 30.02 last 98.34) - open/close 4102 (time 1761322938.25 total 40.04 last 98.88) - open/close 5071 (time 1761322948.26 total 50.04 last 96.84) - open/close 6092 (time 1761322958.27 total 60.05 last 102.02) - open/close 7060 (time 1761322968.27 total 70.05 last 96.76) - open/close 8100 (time 1761322978.27 total 80.05 last 104.00) - open/close 9149 (time 1761322988.27 total 90.05 last 104.90) - open/close 10000 (time 1761322995.52 total 97.31 last 117.31) - open/close 10893 (time 1761323005.53 total 107.32 last 89.24) - open/close 11890 (time 1761323015.55 total 117.33 last 99.56) - open/close 12966 (time 1761323025.55 total 127.34 last 107.51) - open/close 13998 (time 1761323035.56 total 137.35 last 103.12) - open/close 15032 (time 1761323045.57 total 147.35 last 103.36) - open/close 16039 (time 1761323055.58 total 157.36 last 100.56) - open/close 17058 (time 1761323065.58 total 167.37 last 101.86) - open/close 18100 (time 1761323075.59 total 177.37 last 104.14) - open/close 19225 (time 1761323085.59 total 187.38 last 112.45) - open/close 20000 (time 1761323092.59 total 194.37 last 110.78) - open/close 21010 (time 1761323102.60 total 204.38 last 100.93) - open/close 22156 (time 1761323112.60 total 214.39 last 114.54) - open/close 23184 (time 1761323122.60 total 224.39 last 102.77) - open/close 24254 (time 1761323132.61 total 234.39 last 106.98) - open/close 25280 (time 1761323142.61 total 244.39 last 102.57) - open/close 26422 (time 1761323152.61 total 254.40 last 114.13) - open/close 27552 (time 1761323162.62 total 264.41 last 112.93) - open/close 28714 (time 1761323172.63 total 274.41 last 116.15) - open/close 29804 (time 1761323182.63 total 284.42 last 108.90) - open/close 31058 (time 1761323192.64 total 294.43 last 125.27) - open/close 32067 (time 1761323202.65 total 304.43 last 100.87) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 309.33 seconds: 105.37 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 2393 (time 1761323219.66 total 10.00 last 239.21) - /unlink 4670 (time 1761323229.66 total 20.01 last 227.60) - /unlink 6632 (time 1761323239.66 total 30.01 last 196.18) - /unlink 8778 (time 1761323249.66 total 40.01 last 214.58) - /unlink 10000 (time 1761323254.69 total 45.04 last 243.07) - /unlink 12450 (time 1761323264.69 total 55.04 last 244.98) - /unlink 15148 (time 1761323274.69 total 65.04 last 269.79) - /unlink 17035 (time 1761323284.70 total 75.04 last 188.66) - /unlink 19176 (time 1761323294.70 total 85.04 last 214.09) - /unlink 20000 (time 1761323297.86 total 88.21 last 260.01) - /unlink 22469 (time 1761323307.87 total 98.21 last 246.80) - /unlink 24680 (time 1761323317.87 total 108.22 last 221.09) - /unlink 27031 (time 1761323327.87 total 118.22 last 235.02) - /unlink 29515 (time 1761323337.87 total 128.22 last 248.36) - /unlink 31791 (time 1761323347.87 total 138.22 last 227.59) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 142.43 seconds: 228.84 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg122-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing unload_modules_local oleg122-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg122-server: LNET ready to unload oleg122-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 PASS 106 (1365s) Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 oleg122-server: oleg122-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: detected 4 online CPUs by sysfs oleg122-server: Force libcfs to create 2 CPU partitions oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1526 sec ========== 12:30:43 (1761323443)