-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 08:51:48 EDT 2025 oleg450-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg450-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg450-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg450-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg450-server oleg450-server: oleg450-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg450-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg450-server: detected 4 online CPUs by sysfs oleg450-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg450-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg450-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg450-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg450-server: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg450-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg450-server: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg450-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg450-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:53:55 (1761310435) Stopping clients: oleg450-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg450-client.virtnet /mnt/lustre2 (opts:-f) oleg450-server: oleg450-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg450-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg450-server: detected 4 online CPUs by sysfs oleg450-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg450-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg450-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg450-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 2286 (time 1761310492.41 total 10.00 last 228.56) - open/close 4753 (time 1761310502.42 total 20.01 last 246.57) - open/close 6751 (time 1761310512.42 total 30.01 last 199.76) - open/close 9409 (time 1761310522.42 total 40.01 last 265.75) - open/close 10000 (time 1761310524.55 total 42.14 last 276.94) - open/close 14024 (time 1761310534.55 total 52.14 last 402.39) - open/close 19088 (time 1761310544.56 total 62.15 last 506.34) - open/close 20000 (time 1761310546.66 total 64.25 last 433.84) - open/close 23528 (time 1761310556.66 total 74.25 last 352.75) - open/close 28139 (time 1761310566.66 total 84.25 last 461.03) - open/close 30000 (time 1761310570.87 total 88.46 last 441.76) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 94.52 seconds: 344.85 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32878 18 100% /mnt/lustre[OST:0] filesystem_summary: 33142 33124 18 100% /mnt/lustre - /unlink 7360 (time 1761310588.29 total 10.00 last 735.96) - /unlink 10000 (time 1761310591.38 total 13.09 last 854.61) - /unlink 16704 (time 1761310601.42 total 23.13 last 667.88) - /unlink 20000 (time 1761310605.29 total 27.00 last 850.98) - /unlink 24964 (time 1761310615.29 total 37.00 last 496.39) - /unlink 30000 (time 1761310624.33 total 46.04 last 556.90) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 50.80 seconds: 641.61 ops/second Waiting for MDT destroys to complete - open/close 4359 (time 1761310667.24 total 10.00 last 435.87) - open/close 8082 (time 1761310677.24 total 20.00 last 372.20) - open/close 10000 (time 1761310684.62 total 27.38 last 260.00) - open/close 14032 (time 1761310694.62 total 37.38 last 403.20) - open/close 17789 (time 1761310704.62 total 47.38 last 375.67) - open/close 20000 (time 1761310710.24 total 53.00 last 393.33) - open/close 24340 (time 1761310720.25 total 63.00 last 433.94) - open/close 28237 (time 1761310730.25 total 73.00 last 389.68) - open/close 30000 (time 1761310735.43 total 78.19 last 339.78) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 85.96 seconds: 379.19 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 7690 (time 1761310754.77 total 10.00 last 768.99) - /unlink 10000 (time 1761310757.85 total 13.09 last 748.78) - /unlink 16536 (time 1761310767.85 total 23.09 last 653.55) - /unlink 20000 (time 1761310772.57 total 27.80 last 734.26) - /unlink 28692 (time 1761310782.57 total 37.80 last 869.19) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 41.91 seconds: 777.77 ops/second Waiting for MDT destroys to complete - open/close 902 (time 1761310827.87 total 10.00 last 90.17) - open/close 1890 (time 1761310837.88 total 20.01 last 98.69) - open/close 2787 (time 1761310847.90 total 30.03 last 89.57) - open/close 3909 (time 1761310857.90 total 40.03 last 112.16) - open/close 5792 (time 1761310867.90 total 50.04 last 188.24) - open/close 8672 (time 1761310877.91 total 60.04 last 287.97) - open/close 10000 (time 1761310882.95 total 65.09 last 263.04) - open/close 12537 (time 1761310892.95 total 75.09 last 253.70) - open/close 15101 (time 1761310902.96 total 85.09 last 256.32) - open/close 18179 (time 1761310912.96 total 95.09 last 307.77) - open/close 20000 (time 1761310917.67 total 99.81 last 386.16) - open/close 24233 (time 1761310927.68 total 109.81 last 423.15) - open/close 26834 (time 1761310937.68 total 119.81 last 260.08) - open/close 28802 (time 1761310947.68 total 129.81 last 196.78) - open/close 30000 (time 1761310956.00 total 138.13 last 144.02) - open/close 32241 (time 1761310966.00 total 148.13 last 224.07) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 149.33 seconds: 218.26 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 3726 (time 1761310978.85 total 10.00 last 372.56) - /unlink 7748 (time 1761310988.85 total 20.00 last 402.19) - /unlink 10000 (time 1761310993.47 total 24.62 last 487.15) - /unlink 15366 (time 1761311003.47 total 34.62 last 536.58) - /unlink 20000 (time 1761311009.48 total 40.63 last 771.90) - /unlink 25624 (time 1761311019.48 total 50.63 last 562.30) - /unlink 30000 (time 1761311027.01 total 58.16 last 580.81) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 61.80 seconds: 527.42 ops/second Waiting for MDT destroys to complete - open/close 4170 (time 1761311056.45 total 10.00 last 416.95) - open/close 8043 (time 1761311066.45 total 20.00 last 387.29) - open/close 10000 (time 1761311071.27 total 24.82 last 406.05) - open/close 14789 (time 1761311081.27 total 34.82 last 478.83) - open/close 19759 (time 1761311091.27 total 44.82 last 496.88) - open/close 24742 (time 1761311101.27 total 54.83 last 498.27) - open/close 29473 (time 1761311111.27 total 64.83 last 473.09) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 71.53 seconds: 455.64 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9360 (time 1761311129.36 total 10.00 last 935.98) - /unlink 18883 (time 1761311139.36 total 20.00 last 952.27) - /unlink 28783 (time 1761311149.36 total 30.00 last 989.97) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 34.24 seconds: 951.90 ops/second Waiting for MDT destroys to complete - open/close 4992 (time 1761311196.88 total 10.00 last 499.12) - open/close 10000 (time 1761311206.53 total 19.65 last 519.19) - open/close 14878 (time 1761311216.53 total 29.65 last 487.78) - open/close 19925 (time 1761311226.53 total 39.65 last 504.67) - open/close 24734 (time 1761311236.53 total 49.65 last 480.78) - open/close 29525 (time 1761311246.53 total 59.65 last 479.06) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 65.78 seconds: 495.50 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8407 (time 1761311263.98 total 10.00 last 840.63) - /unlink 18123 (time 1761311273.98 total 20.00 last 971.52) - /unlink 27125 (time 1761311283.98 total 30.00 last 900.13) - /unlink 30000 (time 1761311287.46 total 33.48 last 826.88) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 43.15 seconds: 755.31 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg450-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg450-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing unload_modules_local oleg450-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg450-server: LNET ready to unload oleg450-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 PASS 106 (927s) Stopping clients: oleg450-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg450-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 oleg450-server: oleg450-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg450-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: detected 4 online CPUs by sysfs oleg450-server: Force libcfs to create 2 CPU partitions oleg450-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg450-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1117 sec ========== 09:10:26 (1761311426)