-----============= acceptance-small: conf-sanity ============----- Tue Jun 10 21:31:27 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg120-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 55 63 67 32newtarball skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg120-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg120-server oleg120-server: oleg120-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg120-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:33:07 (1749605587) SKIP: conf-sanity test_33a mixed loopback and real device not working SKIP 33a (1s) == conf-sanity test 33b: Drop cancel during umount ======= 21:33:08 (1749605588) start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg120-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0538064 s, 19.5 MB/s fail_loc=0x80000304 umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) umount lustre on /mnt/lustre..... stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 PASS 33b (41s) == conf-sanity test 33c: Mount ost with a large index number ========================================================== 21:33:49 (1749605629) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity formatting backing filesystem ldiskfs on /dev/loop0 target name lustre:MDT0000 kilobytes 200000 options -J size=8 -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -J size=8 -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/loop0 200000k Writing CONFIGS/mountdata pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Permanent disk data: Target: lustre:OST07c6 Index: 1990 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp:192.168.201.120@tcp sys.timeout=20 formatting backing filesystem ldiskfs on /dev/loop0 target name lustre:OST07c6 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST07c6 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/loop0 200000k Writing CONFIGS/mountdata pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Start fs2mds: mount -t lustre -o localrecov /dev/mapper/fs2mds_flakey /mnt/lustre-fs2mds oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /tmp/lustre-mdt1_2 Started lustre-MDT0000 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1751: OST_INDICES: bad array subscript pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Start fs2ost: mount -t lustre -o localrecov /dev/mapper/fs2ost_flakey /mnt/lustre-fs2ost seq.cli-lustre-OST07c6-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST07c6 /mnt/lustre-fs2ost: 142.4 MiB (149295104 bytes) trimmed mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre Creating new pool qpool1 oleg120-server: Pool lustre.qpool1 created Adding targets to pool oleg120-server: OST lustre-OST07c6_UUID added to pool lustre.qpool1 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.37906 s, 52.6 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.021212 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.386871 s, 51.5 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0263259 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.328776 s, 60.6 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.348144 s, 57.2 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0188598 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.33085 s, 60.2 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0167742 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.593411 s, 33.6 MB/s Destroy the created pools: qpool1 lustre.qpool1 oleg120-server: OST lustre-OST07c6_UUID removed from pool lustre.qpool1 oleg120-server: Pool lustre.qpool1 destroyed umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-fs2ost (opts:-f) on oleg120-server Stopping /mnt/lustre-fs2mds (opts:-f) on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local modules unloaded. PASS 33c (125s) == conf-sanity test 33d: Don't panic when enable project quota ========================================================== 21:35:54 (1749605754) start mds service on oleg120-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: No such file or directory oleg120-server: Is the MGS specification correct? oleg120-server: Is the filesystem name correct? oleg120-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 Start of /dev/mapper/mds1_flakey on mds1 failed 2 conf-sanity test_33d: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3721:test_33d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3731:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_33d.*.1749605767.log rsync: chown "/tmp/testlogs/.conf-sanity.test_33d.debug_log.oleg120-server.1749605767.log.Yt1g9e" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_33d.dmesg.oleg120-server.1749605767.log.Gi5zW9" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 33d (14s) == conf-sanity test 34a: umount with opened file should be fail ========================================================== 21:36:08 (1749605768) start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg120-client: oleg120-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_34a: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3734:test_34a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3745:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_34a.*.1749605771.log rsync: chown "/tmp/testlogs/.conf-sanity.test_34a.debug_log.oleg120-server.1749605771.log.VqS8Q4" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_34a.dmesg.oleg120-server.1749605771.log.ZuLb38" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 34a (3s) == conf-sanity test 34b: force umount with failed mds should be normal ========================================================== 21:36:11 (1749605771) start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg120-client: oleg120-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_34b: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3748:test_34b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3756:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_34b.*.1749605775.log rsync: chown "/tmp/testlogs/.conf-sanity.test_34b.debug_log.oleg120-server.1749605775.log.bNTiHq" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_34b.dmesg.oleg120-server.1749605775.log.kAAi5F" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 34b (5s) == conf-sanity test 34c: force umount with failed ost should be normal ========================================================== 21:36:16 (1749605776) start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg120-client: oleg120-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_34c: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3759:test_34c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3767:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_34c.*.1749605779.log rsync: chown "/tmp/testlogs/.conf-sanity.test_34c.debug_log.oleg120-server.1749605779.log.Joovkf" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_34c.dmesg.oleg120-server.1749605779.log.BwJV8D" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 34c (3s) == conf-sanity test 35a: Reconnect to the last active server first ========================================================== 21:36:19 (1749605779) start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg120-client: oleg120-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_35a: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3770:test_35a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3828:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_35a.*.1749605782.log rsync: chown "/tmp/testlogs/.conf-sanity.test_35a.debug_log.oleg120-server.1749605782.log.ujqpr9" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_35a.dmesg.oleg120-server.1749605782.log.lDXr7H" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 35a (4s) == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 21:36:23 (1749605783) start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg120-client: oleg120-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_35b: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3832:test_35b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3919:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_35b.*.1749605786.log rsync: chown "/tmp/testlogs/.conf-sanity.test_35b.debug_log.oleg120-server.1749605786.log.v5scf0" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_35b.dmesg.oleg120-server.1749605786.log.eStXhI" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 35b (4s) == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 21:36:27 (1749605787) SKIP: conf-sanity test_36 remote OST SKIP 36 (0s) == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 21:36:27 (1749605787) MDS : /dev/mapper/mds1_flakey SYMLINK : /tmp/sym_mdt.img mount symlink device - /tmp/sym_mdt.img Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions mount_op=oleg120-server: mount.lustre: mount -t lustre /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists retries left: 0 oleg120-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg120-client: oleg120-server: ssh exited with exit code 17 arg[0] = /sbin/mount.lustre arg[1] = -v arg[2] = -o arg[3] = rw,localrecov arg[4] = /dev/mapper/mds1_flakey arg[5] = /mnt/lustre-mds1 source = /dev/mapper/mds1_flakey (/dev/mapper/mds1_flakey), target = /mnt/lustre-mds1 options(13/4096) = rw,localrecov checking for existing Lustre data: found mounting device /dev/mapper/mds1_flakey at /mnt/lustre-mds1, flags=0x1000000 options=user_xattr,errors=remount-ro,localrecov,osd=osd-ldiskfs,mgs,param=sys.timeout=20,param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity,svname=lustre-MDT0000,device=/dev/mapper/mds1_flakey oleg120-server: umount: /mnt/lustre-mds1: not mounted pdsh@oleg120-client: oleg120-server: ssh exited with exit code 32 conf-sanity test_37: @@@@@@ FAIL: mount symlink /tmp/sym_mdt.img failed! rc=17 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:4040:test_37() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:4042:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_37.*.1749605792.log rsync: chown "/tmp/testlogs/.conf-sanity.test_37.debug_log.oleg120-server.1749605792.log.0tOjaR" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_37.dmesg.oleg120-server.1749605792.log.UNsk5N" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 37 (5s) == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:36:32 (1749605792) start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg120-client: oleg120-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_38: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:4047:test_38() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:4120:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_38.*.1749605795.log rsync: chown "/tmp/testlogs/.conf-sanity.test_38.debug_log.oleg120-server.1749605795.log.JFSvOh" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_38.dmesg.oleg120-server.1749605795.log.Y2XFxn" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 38 (4s) == conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ========================================================== 21:36:36 (1749605796) start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg120-client: oleg120-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 conf-sanity test_39: @@@@@@ FAIL: MDT start failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:220:setup() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:4126:test_39() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:4131:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_39.*.1749605798.log rsync: chown "/tmp/testlogs/.conf-sanity.test_39.debug_log.oleg120-server.1749605798.log.uSqWuM" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_39.dmesg.oleg120-server.1749605798.log.NrO910" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 39 (3s) == conf-sanity test 40: race during service thread startup ========================================================== 21:36:39 (1749605799) start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid fail_loc=0x80000706 start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg120-client: oleg120-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 umount lustre on /mnt/lustre..... stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server stop mds service on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: rmmod: ERROR: Module osp is in use pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 PASS 40 (74s) == conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================================== 21:37:53 (1749605873) start mds service on oleg120-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions Start mds1: mount -t lustre -o localrecov -o nosvc -n /dev/mapper/mds1_flakey /mnt/lustre-mds1 nomtab: 1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all Start /dev/mapper/mds1_flakey without service Started lustre-MDT0000 start mds service on oleg120-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov -o nomgs,force /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists pdsh@oleg120-client: oleg120-server: ssh exited with exit code 17 force: 1 Start of /dev/mapper/mds1_flakey on mds1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre