-----============= acceptance-small: conf-sanity ============----- Wed May 14 17:08:16 EDT 2025 oleg358-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg358-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg358-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg358-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg358-server oleg358-server: oleg358-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: detected 4 online CPUs by sysfs oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg358-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg358-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg358-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg358-server: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg358-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg358-server: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg358-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg358-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:09:48 (1747256988) Stopping clients: oleg358-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg358-client.virtnet /mnt/lustre2 (opts:-f) oleg358-server: oleg358-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: detected 4 online CPUs by sysfs oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg358-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg358-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg358-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock oleg358-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1976 (time 1747257041.90 total 10.00 last 197.54) - open/close 4463 (time 1747257051.91 total 20.01 last 248.65) - open/close 7795 (time 1747257061.91 total 30.01 last 333.20) - open/close 10000 (time 1747257069.68 total 37.78 last 283.55) - open/close 13099 (time 1747257079.69 total 47.78 last 309.84) - open/close 15618 (time 1747257089.69 total 57.78 last 251.88) - open/close 18573 (time 1747257099.69 total 67.79 last 295.47) - open/close 20000 (time 1747257105.96 total 74.06 last 227.52) - open/close 22497 (time 1747257115.96 total 84.06 last 249.65) - open/close 24975 (time 1747257125.96 total 94.06 last 247.79) - open/close 27596 (time 1747257135.96 total 104.06 last 262.01) - open/close 29796 (time 1747257145.97 total 114.07 last 219.92) - open/close 32292 (time 1747257155.97 total 124.07 last 249.56) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 125.19 seconds: 260.35 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4236 (time 1747257168.20 total 10.00 last 423.54) - /unlink 9412 (time 1747257178.20 total 20.00 last 517.56) - /unlink 14435 (time 1747257188.20 total 30.00 last 502.24) - /unlink 19585 (time 1747257198.20 total 40.00 last 514.92) - /unlink 25273 (time 1747257208.20 total 50.01 last 568.80) - /unlink 30000 (time 1747257215.88 total 57.68 last 615.93) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 61.66 seconds: 528.59 ops/second Waiting for MDT destroys to complete - open/close 2760 (time 1747257245.66 total 10.00 last 275.97) - open/close 5676 (time 1747257255.67 total 20.01 last 291.48) - open/close 8404 (time 1747257265.67 total 30.01 last 272.74) - open/close 10000 (time 1747257271.52 total 35.86 last 272.57) - open/close 13820 (time 1747257281.53 total 45.87 last 381.89) - open/close 16648 (time 1747257291.53 total 55.87 last 282.76) - open/close 19594 (time 1747257301.53 total 65.87 last 294.53) - open/close 22732 (time 1747257311.53 total 75.87 last 313.80) - open/close 26146 (time 1747257321.53 total 85.87 last 341.31) - open/close 29410 (time 1747257331.54 total 95.88 last 326.31) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 104.78 seconds: 311.08 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 7295 (time 1747257351.42 total 10.00 last 729.43) - /unlink 10000 (time 1747257355.36 total 13.94 last 686.88) - /unlink 16425 (time 1747257365.36 total 23.94 last 642.40) - /unlink 20000 (time 1747257371.19 total 29.77 last 613.00) - /unlink 27027 (time 1747257381.19 total 39.77 last 702.63) - /unlink 30000 (time 1747257385.92 total 44.50 last 628.88) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 48.72 seconds: 669.05 ops/second Waiting for MDT destroys to complete - open/close 3291 (time 1747257410.38 total 10.00 last 329.03) - open/close 6826 (time 1747257420.38 total 20.00 last 353.49) - open/close 10000 (time 1747257429.74 total 29.36 last 339.05) - open/close 13938 (time 1747257439.74 total 39.36 last 393.79) - open/close 18374 (time 1747257449.74 total 49.36 last 443.58) - open/close 20000 (time 1747257453.47 total 53.09 last 436.37) - open/close 23636 (time 1747257463.47 total 63.09 last 363.55) - open/close 27035 (time 1747257473.47 total 73.09 last 339.89) - open/close 30000 (time 1747257482.22 total 81.84 last 338.94) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 88.76 seconds: 367.23 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 7047 (time 1747257500.07 total 10.00 last 704.69) - /unlink 10000 (time 1747257504.02 total 13.95 last 746.81) - /unlink 17740 (time 1747257514.02 total 23.95 last 774.00) - /unlink 20000 (time 1747257516.87 total 26.80 last 795.34) - /unlink 28465 (time 1747257526.87 total 36.80 last 846.44) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 41.43 seconds: 786.76 ops/second Waiting for MDT destroys to complete - open/close 4797 (time 1747257580.90 total 10.00 last 479.65) - open/close 9863 (time 1747257590.90 total 20.00 last 506.45) - open/close 14975 (time 1747257600.90 total 30.00 last 511.18) - open/close 20000 (time 1747257609.89 total 38.99 last 559.08) - open/close 24950 (time 1747257619.89 total 48.99 last 494.97) - open/close 30000 (time 1747257629.48 total 58.59 last 526.44) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 63.15 seconds: 516.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1747257644.71 total 9.84 last 1016.38) - /unlink 19829 (time 1747257654.71 total 19.84 last 982.81) - /unlink 29050 (time 1747257664.71 total 29.84 last 922.01) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 34.02 seconds: 958.15 ops/second Waiting for MDT destroys to complete - open/close 6389 (time 1747257701.52 total 10.00 last 638.81) - open/close 10000 (time 1747257707.12 total 15.60 last 645.38) - open/close 16356 (time 1747257717.12 total 25.60 last 635.59) - open/close 20000 (time 1747257722.53 total 31.01 last 672.95) - open/close 26742 (time 1747257732.53 total 41.01 last 674.16) - open/close 30000 (time 1747257737.50 total 45.98 last 655.86) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 50.10 seconds: 650.63 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1747257751.90 total 9.50 last 1053.17) - /unlink 20000 (time 1747257761.66 total 19.26 last 1024.07) - /unlink 30000 (time 1747257770.85 total 28.45 last 1088.45) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 30.84 seconds: 1056.81 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg358-server LNET ready to unload unloading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local oleg358-server: LNET ready to unload modules unloaded. pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 PASS 106 (827s) Stopping clients: oleg358-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg358-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 oleg358-server: oleg358-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: detected 4 online CPUs by sysfs oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg358-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 938 sec =========== 17:23:55 (1747257835)