-----============= acceptance-small: conf-sanity ============----- Thu Dec 5 06:15:14 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg336-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg336-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg336-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg336-server oleg336-server: oleg336-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: detected 4 online CPUs by sysfs oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg336-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg336-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg336-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg336-server: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg336-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg336-server: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg336-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 06:16:44 (1733397404) Stopping clients: oleg336-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg336-client.virtnet /mnt/lustre2 (opts:-f) oleg336-server: oleg336-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: detected 4 online CPUs by sysfs oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg336-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg336-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg336-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4989 (time 1733397445.57 total 10.00 last 498.89) - open/close 10000 (time 1733397454.46 total 18.89 last 563.81) - open/close 15441 (time 1733397464.46 total 28.89 last 544.04) - open/close 20000 (time 1733397472.30 total 36.72 last 581.96) - open/close 25705 (time 1733397482.30 total 46.72 last 570.43) - open/close 30000 (time 1733397489.38 total 53.81 last 606.34) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 58.27 seconds: 558.82 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9805 (time 1733397504.69 total 10.00 last 980.42) - /unlink 19254 (time 1733397514.69 total 20.00 last 944.89) - /unlink 26771 (time 1733397524.69 total 30.00 last 751.70) - /unlink 30000 (time 1733397528.40 total 33.71 last 870.80) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 36.96 seconds: 880.84 ops/second Waiting for MDT destroys to complete oleg336-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg336-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 - open/close 5191 (time 1733397570.98 total 10.00 last 518.99) - open/close 10000 (time 1733397579.41 total 18.43 last 570.65) - open/close 16267 (time 1733397589.41 total 28.43 last 626.61) - open/close 20000 (time 1733397595.48 total 34.50 last 615.48) - open/close 26187 (time 1733397605.48 total 44.50 last 618.64) - open/close 30000 (time 1733397611.44 total 50.46 last 639.41) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 54.48 seconds: 597.62 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1733397625.48 total 9.21 last 1085.73) - /unlink 19733 (time 1733397635.48 total 19.21 last 973.25) - /unlink 29223 (time 1733397645.48 total 29.21 last 948.94) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.65 seconds: 997.13 ops/second Waiting for MDT destroys to complete oleg336-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg336-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 - open/close 5636 (time 1733397688.40 total 10.00 last 563.53) - open/close 10000 (time 1733397696.13 total 17.73 last 564.92) - open/close 15539 (time 1733397706.13 total 27.73 last 553.86) - open/close 20000 (time 1733397714.19 total 35.79 last 553.48) - open/close 25603 (time 1733397724.19 total 45.79 last 560.25) - open/close 30000 (time 1733397732.09 total 53.69 last 556.77) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 58.37 seconds: 557.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9618 (time 1733397747.64 total 10.00 last 961.77) - /unlink 18705 (time 1733397757.64 total 20.00 last 908.66) - /unlink 28425 (time 1733397767.64 total 30.01 last 971.26) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.08 seconds: 955.50 ops/second Waiting for MDT destroys to complete oleg336-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg336-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 - open/close 5590 (time 1733397808.68 total 10.00 last 558.96) - open/close 10000 (time 1733397816.40 total 17.72 last 570.93) - open/close 15596 (time 1733397826.40 total 27.73 last 559.56) - open/close 20000 (time 1733397834.33 total 35.65 last 555.59) - open/close 25493 (time 1733397844.33 total 45.65 last 549.24) - open/close 30000 (time 1733397852.59 total 53.92 last 545.50) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 58.63 seconds: 555.36 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9686 (time 1733397868.19 total 10.00 last 968.52) - /unlink 19423 (time 1733397878.19 total 20.00 last 973.51) - /unlink 29222 (time 1733397888.19 total 30.00 last 979.82) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.28 seconds: 978.31 ops/second Waiting for MDT destroys to complete oleg336-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg336-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 - open/close 5471 (time 1733397929.79 total 10.00 last 547.08) - open/close 10000 (time 1733397937.85 total 18.05 last 562.34) - open/close 15456 (time 1733397947.85 total 28.06 last 545.50) - open/close 20000 (time 1733397956.07 total 36.28 last 552.62) - open/close 25551 (time 1733397966.07 total 46.28 last 555.10) - open/close 30000 (time 1733397974.33 total 54.53 last 538.89) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.36 seconds: 548.56 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9770 (time 1733397990.06 total 10.00 last 976.93) - /unlink 19557 (time 1733398000.06 total 20.00 last 978.64) - /unlink 29453 (time 1733398010.06 total 30.00 last 989.60) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.13 seconds: 982.93 ops/second Waiting for MDT destroys to complete oleg336-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg336-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server unloading modules via unload_modules_local on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 PASS 106 (676s) Stopping clients: oleg336-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg336-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 oleg336-server: oleg336-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: detected 4 online CPUs by sysfs oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg336-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 783 sec =========== 06:28:18 (1733398098) === conf-sanity: start cleanup 06:28:19 (1733398099) === === conf-sanity: finish cleanup 06:28:19 (1733398099) ===