-----============= acceptance-small: conf-sanity ============----- Tue Jun 10 21:31:25 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg408-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 21e 32b 32c 32newtarball skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg408-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server == conf-sanity test 0: single mount setup ================ 21:32:51 (1749605571) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 PASS 0 (39s) == conf-sanity test 1: start up ost twice (should return errors) ========================================================== 21:33:31 (1749605611) start mds service on oleg408-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: No such file or directory oleg408-server: Is the MGS specification correct? oleg408-server: Is the filesystem name correct? oleg408-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 Start of /dev/mapper/mds1_flakey on mds1 failed 2 conf-sanity test_1: @@@@@@ FAIL: MDS start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:306:test_1() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:314:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_1.*.1749605622.log rsync: chown "/tmp/testlogs/.conf-sanity.test_1.debug_log.oleg408-server.1749605622.log.5HNzrZ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_1.dmesg.oleg408-server.1749605622.log.8kJ9KN" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 1 (11s) == conf-sanity test 2: start up mds twice (should return err) ========================================================== 21:33:43 (1749605623) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_2: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:317:test_2() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:325:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_2.*.1749605625.log rsync: chown "/tmp/testlogs/.conf-sanity.test_2.debug_log.oleg408-server.1749605625.log.oa72nE" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_2.dmesg.oleg408-server.1749605625.log.GdMJ2B" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 2 (4s) == conf-sanity test 3: mount client twice (should return err) ========================================================== 21:33:46 (1749605626) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_3: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:328:test_3() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:334:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_3.*.1749605629.log rsync: chown "/tmp/testlogs/.conf-sanity.test_3.debug_log.oleg408-server.1749605629.log.NBpHhj" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_3.dmesg.oleg408-server.1749605629.log.MbJ0iq" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 3 (4s) == conf-sanity test 4: force cleanup ost, then cleanup === 21:33:50 (1749605630) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_4: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:337:test_4() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:348:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_4.*.1749605632.log rsync: chown "/tmp/testlogs/.conf-sanity.test_4.debug_log.oleg408-server.1749605632.log.R6M9t5" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_4.dmesg.oleg408-server.1749605632.log.NTOmPl" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 4 (3s) == conf-sanity test 5a: force cleanup mds, then cleanup == 21:33:53 (1749605633) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_5a: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:351:test_5a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:400:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_5a.*.1749605636.log rsync: chown "/tmp/testlogs/.conf-sanity.test_5a.debug_log.oleg408-server.1749605636.log.RBJW3F" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_5a.dmesg.oleg408-server.1749605636.log.8dFlj5" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 5a (4s) == conf-sanity test 5b: Try to start a client with no MGS (should return errs) ========================================================== 21:33:57 (1749605637) start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? umount lustre on /mnt/lustre..... stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server stop mds service on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: rmmod: ERROR: Module osp is in use pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 PASS 5b (225s) == conf-sanity test 5c: cleanup after failed mount (bug 2712) (should return errs) ========================================================== 21:37:42 (1749605862) start mds service on oleg408-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_5c: @@@@@@ FAIL: MDS start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:434:test_5c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:445:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_5c.*.1749605870.log rsync: chown "/tmp/testlogs/.conf-sanity.test_5c.debug_log.oleg408-server.1749605870.log.07WVTg" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_5c.dmesg.oleg408-server.1749605870.log.mdDhAy" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 5c (8s) == conf-sanity test 5d: mount with ost down ============== 21:37:50 (1749605870) start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_5d: @@@@@@ FAIL: MDS start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:452:test_5d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:460:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_5d.*.1749605929.log rsync: chown "/tmp/testlogs/.conf-sanity.test_5d.debug_log.oleg408-server.1749605929.log.Uvr7vq" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_5d.dmesg.oleg408-server.1749605929.log.AQZgGe" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 5d (60s) == conf-sanity test 5e: delayed connect, don't crash (bug 10268) ========================================================== 21:38:50 (1749605930) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_5e: @@@@@@ FAIL: MDS start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:466:test_5e() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:477:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_5e.*.1749605933.log rsync: chown "/tmp/testlogs/.conf-sanity.test_5e.debug_log.oleg408-server.1749605933.log.X9gpye" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_5e.dmesg.oleg408-server.1749605933.log.Z5HXVb" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 5e (4s) == conf-sanity test 5f: mds down, cleanup after failed mount (bug 2712) ========================================================== 21:38:54 (1749605934) SKIP: conf-sanity test_5f needs separate mgs and mds SKIP 5f (0s) == conf-sanity test 5g: handle missing debugfs =========== 21:38:54 (1749605934) modprobe: FATAL: Module lustre not found. none /sys/kernel/debug debugfs rw,relatime 0 0 PASS 5g (2s) == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 21:38:56 (1749605936) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_5h: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:528:test_5h() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:538:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_5h.*.1749605939.log rsync: chown "/tmp/testlogs/.conf-sanity.test_5h.debug_log.oleg408-server.1749605939.log.XtWzOi" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_5h.dmesg.oleg408-server.1749605939.log.7ejMwv" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 5h (3s) == conf-sanity test 5i: start mdt failure at mdt_quota_init() ========================================================== 21:38:59 (1749605939) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_5i: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:544:test_5i() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:554:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_5i.*.1749605942.log rsync: chown "/tmp/testlogs/.conf-sanity.test_5i.debug_log.oleg408-server.1749605942.log.mCOj8R" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_5i.dmesg.oleg408-server.1749605942.log.T2fG0d" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 5i (4s) == conf-sanity test 5j: verify libcfs doesn't crash when setting debug flags ========================================================== 21:39:03 (1749605943) unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: rmmod: ERROR: Module lustre is in use pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 modules unloaded. debug=all pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 PASS 5j (3s) == conf-sanity test 6: manual umount, then mount again === 21:39:06 (1749605946) start mds service on oleg408-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_6: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:566:test_6() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:572:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_6.*.1749605953.log rsync: chown "/tmp/testlogs/.conf-sanity.test_6.debug_log.oleg408-server.1749605953.log.OeSBpB" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_6.dmesg.oleg408-server.1749605953.log.y1n4Bq" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 6 (8s) == conf-sanity test 7: manual umount, then cleanup ======= 21:39:14 (1749605954) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_7: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:575:test_7() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:579:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_7.*.1749605957.log rsync: chown "/tmp/testlogs/.conf-sanity.test_7.debug_log.oleg408-server.1749605957.log.5FN8hf" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_7.dmesg.oleg408-server.1749605957.log.uyfsJd" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 7 (4s) == conf-sanity test 8: double mount setup ================ 21:39:18 (1749605958) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_8: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:582:test_8() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:588:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_8.*.1749605960.log rsync: chown "/tmp/testlogs/.conf-sanity.test_8.debug_log.oleg408-server.1749605960.log.9aDCRS" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_8.dmesg.oleg408-server.1749605960.log.RdSTh0" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 8 (3s) == conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================================== 21:39:21 (1749605961) start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg408-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 conf-sanity test_9: @@@@@@ FAIL: OST start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:591:test_9() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:614:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_9.*.1749605964.log rsync: chown "/tmp/testlogs/.conf-sanity.test_9.debug_log.oleg408-server.1749605964.log.b9L5kx" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_9.dmesg.oleg408-server.1749605964.log.P0wF3N" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 9 (4s) == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 21:39:25 (1749605965) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_10a: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:617:test_10a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:635:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_10a.*.1749605967.log rsync: chown "/tmp/testlogs/.conf-sanity.test_10a.debug_log.oleg408-server.1749605967.log.1t1jR1" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_10a.dmesg.oleg408-server.1749605967.log.gozBpr" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 10a (3s) == conf-sanity test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 21:39:28 (1749605968) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_17: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:648:test_17() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:669:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_17.*.1749605971.log rsync: chown "/tmp/testlogs/.conf-sanity.test_17.debug_log.oleg408-server.1749605971.log.8aC4DB" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_17.dmesg.oleg408-server.1749605971.log.7BLsX9" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 17 (3s) == conf-sanity test 18: check mkfs creates large journals ========================================================== 21:39:31 (1749605971) use device /dev/mapper/mds1_flakey with MIN=2000000 mount mds with large journal... Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server mount lustre system... start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre setup single mount lustre success check journal size... oleg408-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg408-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps Success: mkfs creates large journals. Size: 78M umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local modules unloaded. Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server PASS 18 (132s) == conf-sanity test 19a: start/stop MDS without OSTs ===== 21:41:43 (1749606103) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server PASS 19a (17s) == conf-sanity test 19b: start/stop OSTs without MDS ===== 21:42:00 (1749606120) start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server PASS 19b (59s) == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 21:42:59 (1749606179) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre setup single mount lustre success mount lustre with opts remount,ro on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o remount,ro 192.168.204.108@tcp:/lustre /mnt/lustre touch: cannot touch '/mnt/lustre/f20.conf-sanity': Read-only file system mount lustre with opts remount,rw on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o remount,rw 192.168.204.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server PASS 20 (41s) == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 21:43:40 (1749606220) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server PASS 21a (33s) == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 21:44:13 (1749606253) start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server PASS 21b (92s) == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 21:45:45 (1749606345) start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost2 service on oleg408-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop ost2 service on oleg408-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 PASS 21c (108s) == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 21:47:33 (1749606453) SKIP: conf-sanity test_21d need separate mgs device SKIP 21d (1s) SKIP: conf-sanity test_21e skipping excluded test 21e == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 21:47:35 (1749606455) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg408-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88013004f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88013004f000.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 PASS 22 (71s) == conf-sanity test 23a: interrupt client during recovery mount delay ========================================================== 21:48:46 (1749606526) start mds service on oleg408-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: No such file or directory oleg408-server: Is the MGS specification correct? oleg408-server: Is the filesystem name correct? oleg408-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 Start of /dev/mapper/mds1_flakey on mds1 failed 2 conf-sanity test_23a: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:918:test_23a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:969:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_23a.*.1749606538.log rsync: chown "/tmp/testlogs/.conf-sanity.test_23a.debug_log.oleg408-server.1749606538.log.578hqD" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_23a.dmesg.oleg408-server.1749606538.log.m0ikk9" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 23a (12s) == conf-sanity test 23b: Simulate -EINTR during mount ==== 21:48:58 (1749606538) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_23b: @@@@@@ FAIL: MDS start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:972:test_23b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:979:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_23b.*.1749606542.log rsync: chown "/tmp/testlogs/.conf-sanity.test_23b.debug_log.oleg408-server.1749606542.log.1BPq5S" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_23b.dmesg.oleg408-server.1749606542.log.3OkAUz" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 23b (5s) == conf-sanity test 24a: Multiple MDTs on a single node == 21:49:03 (1749606543) SKIP: conf-sanity test_24a mixed loopback and real device not working SKIP 24a (1s) == conf-sanity test 24b: Multiple MGSs on a single node (should return err) ========================================================== 21:49:04 (1749606544) SKIP: conf-sanity test_24b mixed loopback and real device not working SKIP 24b (1s) == conf-sanity test 25: Verify modules are referenced ==== 21:49:05 (1749606545) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? conf-sanity test_25: @@@@@@ FAIL: check_mount failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1077:test_25() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1082:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_25.*.1749606767.log rsync: chown "/tmp/testlogs/.conf-sanity.test_25.debug_log.oleg408-server.1749606767.log.wlkugv" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_25.dmesg.oleg408-server.1749606767.log.GamJSH" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 25 (223s) == conf-sanity test 26: MDT startup failure cleans LOV (should return errs) ========================================================== 21:52:48 (1749606768) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions fail_loc=0x80000135 start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: No such file or directory oleg408-server: Is the MGS specification correct? oleg408-server: Is the filesystem name correct? oleg408-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 Start of /dev/mapper/mds1_flakey on mds1 failed 2 start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_26: @@@@@@ FAIL: Unable to start MDS Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1095:test_26() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1099:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_26.*.1749606778.log rsync: chown "/tmp/testlogs/.conf-sanity.test_26.debug_log.oleg408-server.1749606778.log.OrUr4Q" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_26.dmesg.oleg408-server.1749606778.log.vs65bw" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 26 (11s) == conf-sanity test 27a: Reacquire MGS lock if OST started first ========================================================== 21:52:59 (1749606779) umount lustre on /mnt/lustre..... stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server stop mds service on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: rmmod: ERROR: Module osp is in use pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 modules unloaded. start ost1 service on oleg408-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_27a: @@@@@@ FAIL: Unable to start MDS Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1104:test_27a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1112:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_27a.*.1749606863.log rsync: chown "/tmp/testlogs/.conf-sanity.test_27a.debug_log.oleg408-server.1749606863.log.0PFrwb" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_27a.dmesg.oleg408-server.1749606863.log.q22f6s" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 27a (85s) == conf-sanity test 27b: Reacquire MGS lock after failover ========================================================== 21:54:24 (1749606864) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg408-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? Failing mds1 on oleg408-server 21:57:14 (1749607034) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_27b: @@@@@@ FAIL: Restart of mds1 failed! Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:2139:mount_facets() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:4327:facet_failover() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1120:test_27b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1130:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_27b.*.1749607057.log 21:57:38 (1749607058) targets are mounted 21:57:38 (1749607058) facet_failover done rsync: chown "/tmp/testlogs/.conf-sanity.test_27b.debug_log.oleg408-server.1749607057.log.smcWqq" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_27b.dmesg.oleg408-server.1749607057.log.2qNEvU" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] oleg408-server: error: get_param: param_path 'mdt//identity_acquire_expire': No such file or directory pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 oleg408-server: error: get_param: param_path 'mdt//identity_acquire_expire': No such file or directory pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 Setting .mdt.identity_acquire_expire from to 5 oleg408-server: error: conf_param: Unknown error 524 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 12 conf-sanity test_27b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lctl conf_param .mdt.identity_acquire_expire failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6143:set_persistent_param() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6160:set_persistent_param_and_check() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1121:test_27b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1130:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_27b.*.1749607060.log rsync: chown "/tmp/testlogs/.conf-sanity.test_27b.debug_log.oleg408-server.1749607060.log.ctkpfE" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_27b.dmesg.oleg408-server.1749607060.log.yM0Jud" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 27b (197s) == conf-sanity test 28A: permanent parameter setting ===== 21:57:41 (1749607061) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg408-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? error: get_param: param_path 'llite/lustre-*/max_read_ahead_whole_mb': No such file or directory error: get_param: param_path 'llite/lustre-*/max_read_ahead_per_file_mb': No such file or directory ORIG: MAX: ORIG:-3 MAX: error: get_param: param_path 'llite/lustre-*/max_read_ahead_whole_mb': No such file or directory error: get_param: param_path 'llite/lustre-*/max_read_ahead_whole_mb': No such file or directory Setting lustre.llite.max_read_ahead_whole_mb from to -2 oleg408-server: error: conf_param: Unknown error 524 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 12 conf-sanity test_28A: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lctl conf_param lustre.llite.max_read_ahead_whole_mb failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6143:set_persistent_param() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6160:set_persistent_param_and_check() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1149:test_28A() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1165:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_28A.*.1749607229.log rsync: chown "/tmp/testlogs/.conf-sanity.test_28A.debug_log.oleg408-server.1749607229.log.Dz1wqj" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_28A.dmesg.oleg408-server.1749607229.log.APYTb3" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 28A (169s) == conf-sanity test 28a: set symlink parameters permanently with lctl ========================================================== 22:00:30 (1749607230) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg408-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? Setting lustre-OST0000.ost.client_cache_seconds from 30 to 60 oleg408-server: error: conf_param: Unknown error 524 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 12 conf-sanity test_28a: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lctl conf_param lustre-OST0000.ost.client_cache_seconds failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6143:set_persistent_param() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6160:set_persistent_param_and_check() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1196:test_28a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1213:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_28a.*.1749607418.log rsync: chown "/tmp/testlogs/.conf-sanity.test_28a.debug_log.oleg408-server.1749607418.log.R02bsV" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_28a.dmesg.oleg408-server.1749607418.log.JaFUpD" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 28a (189s) == conf-sanity test 28b: verify client-side parameters are set upon mount ========================================================== 22:03:39 (1749607419) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg408-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? conf-sanity test_28b: @@@@@@ FAIL: mount failed with (5) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:5740:remount_client() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1254:test_28b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1263:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_28b.*.1749607770.log jobid_name=%e.%u umount lustre on /mnt/lustre..... rsync: chown "/tmp/testlogs/.conf-sanity.test_28b.debug_log.oleg408-server.1749607770.log.RIjV3I" failed: Operation not permitted (1) stop ost1 service on oleg408-server rsync: chown "/tmp/testlogs/.conf-sanity.test_28b.dmesg.oleg408-server.1749607770.log.uXNIpl" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server stop mds service on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: rmmod: ERROR: Module osp is in use pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 modules unloaded. FAIL 28b (359s) == conf-sanity test 28c: verify filesystem parameters are set upon mount ========================================================== 22:09:38 (1749607778) start mds service on oleg408-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? conf-sanity test_28c: @@@@@@ FAIL: mount failed with (5) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:5740:remount_client() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1294:test_28c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1303:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_28c.*.1749608169.log jobid_name=%e.%u rsync: chown "/tmp/testlogs/.conf-sanity.test_28c.debug_log.oleg408-server.1749608169.log.bp1JfK" failed: Operation not permitted (1) umount lustre on /mnt/lustre..... rsync: chown "/tmp/testlogs/.conf-sanity.test_28c.dmesg.oleg408-server.1749608169.log.a6ARNb" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server stop mds service on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: rmmod: ERROR: Module osp is in use pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 modules unloaded. FAIL 28c (397s) == conf-sanity test 29: permanently remove an OST ======== 22:16:15 (1749608175) start ost2 service on oleg408-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 oleg408-server: mount.lustre: mount /dev/mapper/ost2_flakey at /mnt/lustre-ost2 failed: Connection timed out pdsh@oleg408-client: oleg408-server: ssh exited with exit code 110 oleg408-server: error: set_param: param_path 'seq/cli-lustre': No such file or directory oleg408-server: error: set_param: setting 'seq/cli-lustre'='OST0001-super.width=65536': No such file or directory pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 Start of /dev/mapper/ost2_flakey on ost2 failed 110 conf-sanity test_29: @@@@@@ FAIL: Unable to start OST2 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1308:test_29() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1357:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_29.*.1749608435.log rsync: chown "/tmp/testlogs/.conf-sanity.test_29.debug_log.oleg408-server.1749608435.log.7TZzMH" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_29.dmesg.oleg408-server.1749608435.log.9ZRajn" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 29 (260s) == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 22:20:35 (1749608435) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg408-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? Big config llog error: get_param: param_path 'llite/lustre-*/max_read_ahead_whole_mb': No such file or directory error: get_param: param_path 'llite/lustre-*/max_read_ahead_whole_mb': No such file or directory error: get_param: param_path 'llite/lustre-*/max_read_ahead_whole_mb': No such file or directory Setting lustre.llite.max_read_ahead_whole_mb from to 1 oleg408-server: error: conf_param: Unknown error 524 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 12 conf-sanity test_30a: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lctl conf_param lustre.llite.max_read_ahead_whole_mb failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6143:set_persistent_param() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6160:set_persistent_param_and_check() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1373:test_30a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1402:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_30a.*.1749608603.log rsync: chown "/tmp/testlogs/.conf-sanity.test_30a.debug_log.oleg408-server.1749608603.log.zC5KZX" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_30a.dmesg.oleg408-server.1749608603.log.Fgj1WJ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 30a (169s) == conf-sanity test 30b: Remove failover nids ============ 22:23:24 (1749608604) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg408-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? error: get_param: param_path 'osc/lustre-OST0000-osc-[^M]*/import': No such file or directory Checking peer and local NIDs: 192.168.204.108@tcp Changing 192.168.204.108@tcp to fake NID 192.168.204.209@tcp Setting lustre-OST0000.failover.node from 192.168.204.108@tcp to 192.168.204.209@tcp oleg408-server: error: conf_param: Unknown error 524 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 12 conf-sanity test_30b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lctl conf_param lustre-OST0000.failover.node failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1439:test_30b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1475:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_30b.*.1749608794.log rsync: chown "/tmp/testlogs/.conf-sanity.test_30b.debug_log.oleg408-server.1749608794.log.3Ijb49" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_30b.dmesg.oleg408-server.1749608794.log.uAQeB0" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 30b (191s) == conf-sanity test 31: Connect to non-existent node (shouldn't crash) ========================================================== 22:26:35 (1749608795) mount.lustre: mount 4.3.2.1@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? umount lustre on /mnt/lustre..... stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server stop mds service on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: rmmod: ERROR: Module osp is in use pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 PASS 31 (258s) Failing mds1 on oleg408-server 22:30:55 (1749609055) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg408-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_802a: @@@@@@ FAIL: Restart of mds1 failed! Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:2139:mount_facets() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:4327:facet_failover() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:4446:fail() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:5322:stopall() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:5676:formatall() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:98:reformat() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:12676:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_802a.*.1749609072.log rsync: chown "/tmp/testlogs/.conf-sanity.test_802a.debug_log.oleg408-server.1749609072.log.HU5PvV" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_802a.dmesg.oleg408-server.1749609072.log.xBYdIw" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] 22:31:13 (1749609073) targets are mounted 22:31:13 (1749609073) facet_failover done pdsh@oleg408-client: oleg408-client: ssh exited with exit code 95 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg408-client: oleg408-client: ssh exited with exit code 95 Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3604 sec ========== 22:31:29 (1749609089) conf-sanity: FAIL: test_1 MDS start failed conf-sanity: FAIL: test_2 MDT start failed conf-sanity: FAIL: test_3 MDT start failed conf-sanity: FAIL: test_4 MDT start failed conf-sanity: FAIL: test_5a MDT start failed conf-sanity: FAIL: test_5c MDS start failed conf-sanity: FAIL: test_5d MDS start failed conf-sanity: FAIL: test_5e MDS start failed conf-sanity: FAIL: test_5h MDT start failed conf-sanity: FAIL: test_5i MDT start failed conf-sanity: FAIL: test_6 MDT start failed conf-sanity: FAIL: test_7 MDT start failed conf-sanity: FAIL: test_8 MDT start failed conf-sanity: FAIL: test_9 OST start failed conf-sanity: FAIL: test_10a MDT start failed conf-sanity: FAIL: test_17 MDT start failed conf-sanity: FAIL: test_23a MDT start failed conf-sanity: FAIL: test_23b MDS start failed conf-sanity: FAIL: test_25 check_mount failed conf-sanity: FAIL: test_26 Unable to start MDS conf-sanity: FAIL: test_27a Unable to start MDS conf-sanity: FAIL: test_27b Restart of mds1 failed! conf-sanity: FAIL: test_27b /home/green/git/lustre-release/lustre/utils/lctl conf_param .mdt.identity_acquire_expire failed conf-sanity: FAIL: test_28A /home/green/git/lustre-release/lustre/utils/lctl conf_param lustre.llite.max_read_ahead_whole_mb failed conf-sanity: FAIL: test_28a /home/green/git/lustre-release/lustre/utils/lctl conf_param lustre-OST0000.ost.client_cache_seconds failed conf-sanity: FAIL: test_28b mount failed with (5) conf-sanity: FAIL: test_28c mount failed with (5) conf-sanity: FAIL: test_29 Unable to start OST2 conf-sanity: FAIL: test_30a /home/green/git/lustre-release/lustre/utils/lctl conf_param lustre.llite.max_read_ahead_whole_mb failed conf-sanity: FAIL: test_30b /home/green/git/lustre-release/lustre/utils/lctl conf_param lustre-OST0000.failover.node failed conf-sanity: FAIL: test_802a Restart of mds1 failed! === conf-sanity: start cleanup 22:31:30 (1749609090) === === conf-sanity: finish cleanup 22:31:30 (1749609090) ===