-----============= acceptance-small: conf-sanity ============----- Tue Aug 27 11:41:52 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg319-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg319-server oleg319-server: oleg319-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg319-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg319-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg319-server: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg319-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg319-server: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg319-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg319-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:43:22 (1724773402) Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) oleg319-server: oleg319-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg319-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg319-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3706 (time 1724773442.59 total 10.00 last 370.58) - open/close 8458 (time 1724773452.59 total 20.00 last 475.18) - open/close 10000 (time 1724773455.61 total 23.03 last 509.69) - open/close 16010 (time 1724773465.61 total 33.03 last 600.94) - open/close 20000 (time 1724773471.83 total 39.24 last 641.80) - open/close 25820 (time 1724773481.83 total 49.24 last 581.94) - open/close 30000 (time 1724773489.94 total 57.35 last 515.57) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 62.25 seconds: 523.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9951 (time 1724773505.71 total 10.00 last 995.02) - /unlink 19285 (time 1724773515.72 total 20.00 last 933.33) - /unlink 28631 (time 1724773525.72 total 30.00 last 934.48) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.53 seconds: 943.03 ops/second Waiting for MDT destroys to complete oleg319-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg319-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 - open/close 4717 (time 1724773585.67 total 10.00 last 471.68) - open/close 9311 (time 1724773595.67 total 20.00 last 459.38) - open/close 13250 (time 1724773605.67 total 30.00 last 393.89) - open/close 16268 (time 1724773615.67 total 40.00 last 301.78) - open/close 18785 (time 1724773625.68 total 50.01 last 251.58) - open/close 20000 (time 1724773630.81 total 55.14 last 236.77) - open/close 23388 (time 1724773640.81 total 65.14 last 338.70) - open/close 25764 (time 1724773650.81 total 75.14 last 237.52) - open/close 28446 (time 1724773660.82 total 85.15 last 268.11) - open/close 30000 (time 1724773667.25 total 91.58 last 241.61) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 101.36 seconds: 321.22 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5959 (time 1724773688.15 total 10.00 last 595.79) - /unlink 10000 (time 1724773693.51 total 15.36 last 754.34) - /unlink 18084 (time 1724773703.51 total 25.36 last 808.39) - /unlink 20000 (time 1724773706.20 total 28.05 last 711.59) - /unlink 27929 (time 1724773716.20 total 38.05 last 792.84) - /unlink 30000 (time 1724773718.22 total 40.07 last 1027.64) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 43.45 seconds: 749.41 ops/second Waiting for MDT destroys to complete oleg319-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg319-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 - open/close 3405 (time 1724773760.98 total 10.00 last 340.42) - open/close 7466 (time 1724773770.98 total 20.00 last 406.09) - open/close 10000 (time 1724773777.20 total 26.23 last 407.03) - open/close 13626 (time 1724773787.20 total 36.23 last 362.60) - open/close 16860 (time 1724773797.20 total 46.23 last 323.34) - open/close 20000 (time 1724773806.13 total 55.16 last 351.65) - open/close 23502 (time 1724773816.13 total 65.16 last 350.14) - open/close 26668 (time 1724773826.14 total 75.16 last 316.54) - open/close 29994 (time 1724773836.14 total 85.16 last 332.59) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 93.12 seconds: 349.66 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7304 (time 1724773855.15 total 10.00 last 730.37) - /unlink 10000 (time 1724773858.12 total 12.98 last 906.17) - /unlink 18304 (time 1724773868.12 total 22.98 last 830.39) - /unlink 25706 (time 1724773878.12 total 32.98 last 740.13) - /unlink 30000 (time 1724773884.32 total 39.17 last 692.86) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 42.51 seconds: 765.87 ops/second Waiting for MDT destroys to complete oleg319-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg319-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 - open/close 4488 (time 1724773924.37 total 10.00 last 448.72) - open/close 9124 (time 1724773934.38 total 20.00 last 463.51) - open/close 14000 (time 1724773944.38 total 30.00 last 487.55) - open/close 19271 (time 1724773954.38 total 40.01 last 527.07) - open/close 24077 (time 1724773964.38 total 50.01 last 480.56) - open/close 29429 (time 1724773974.38 total 60.01 last 535.14) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 65.04 seconds: 500.62 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9486 (time 1724773990.32 total 10.00 last 948.52) - /unlink 19096 (time 1724774000.32 total 20.00 last 960.97) - /unlink 28099 (time 1724774010.32 total 30.00 last 900.17) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.15 seconds: 953.51 ops/second Waiting for MDT destroys to complete oleg319-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg319-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 - open/close 5240 (time 1724774062.10 total 10.00 last 523.96) - open/close 10000 (time 1724774071.08 total 18.98 last 529.92) - open/close 14819 (time 1724774081.08 total 28.98 last 481.88) - open/close 20000 (time 1724774089.88 total 37.78 last 588.90) - open/close 25435 (time 1724774099.88 total 47.78 last 543.39) - open/close 30000 (time 1724774108.73 total 56.64 last 515.46) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 61.02 seconds: 533.59 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9530 (time 1724774124.00 total 10.00 last 952.96) - /unlink 19392 (time 1724774134.00 total 20.00 last 986.18) - /unlink 29660 (time 1724774144.00 total 30.00 last 1026.76) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.96 seconds: 988.00 ops/second Waiting for MDT destroys to complete oleg319-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 oleg319-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg319-server unloading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 PASS 106 (788s) Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 oleg319-server: oleg319-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg319-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 895 sec =========== 11:56:48 (1724774208) === conf-sanity: start cleanup 11:56:49 (1724774209) === === conf-sanity: finish cleanup 11:56:49 (1724774209) ===