-----============= acceptance-small: conf-sanity ============----- Wed May 8 18:02:17 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 55 63 67 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg438-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg438-server oleg438-server: oleg438-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg438-server: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg438-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:03:21 (1715205801) pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Permanent disk data: Target: test-33a:MDT0000 Index: 0 Lustre FS: test-33a Mount type: zfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto Writing lustre-mdt1_2/mdt1_2 properties lustre:mgsnode=192.168.204.138@tcp lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=101 lustre:index=0 lustre:fsname=test-33a lustre:svname=test-33a:MDT0000 Permanent disk data: Target: test-33a:OST1fff Index: 8191 Lustre FS: test-33a Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 mgsnode=192.168.204.138@tcp:192.168.204.138@tcp autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1_2/ost1_2 properties lustre:sys.timeout=20 lustre:mgsnode=192.168.204.138@tcp:192.168.204.138@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=8191 lustre:fsname=test-33a lustre:svname=test-33a:OST1fff Starting fs2mds: -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1_2/mdt1_2 Started test-33a-MDT0000 Starting fs2ost: -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-test-33a-OST1fff-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started test-33a-OST1fff ok. /mnt/lustre2/hosts lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 8191 obdidx objid objid group 8191 2 0x2 0x240000400 config_log: test-33a-client cancel catalog test-33a-client log_idx 25: done cancel catalog test-33a-client log_idx 24: done cancel catalog test-33a-client log_idx 23: done del_ost: cancelled 3 catalog entries config_log: test-33a-MDT0000 cancel catalog test-33a-MDT0000 log_idx 20: done cancel catalog test-33a-MDT0000 log_idx 19: done cancel catalog test-33a-MDT0000 log_idx 18: done del_ost: cancelled 3 catalog entries ok. Stopping /mnt/lustre-fs2ost (opts:-f) on oleg438-server Stopping /mnt/lustre-fs2mds (opts:-f) on oleg438-server stop ost1 service on oleg438-server stop mds service on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 umount /mnt/lustre2 ... umount: /mnt/lustre2: not mounted stopping fs2mds ... stopping fs2ost ... PASS 33a (29s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg438-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 33b: Drop cancel during umount ======= 18:03:52 (1715205832) start mds service on oleg438-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0320969 s, 32.7 MB/s fail_loc=0x80000304 umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) umount lustre on /mnt/lustre..... stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 PASS 33b (35s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg438-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 33c: Mount ost with a large index number ========================================================== 18:04:29 (1715205869) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto Writing lustre-mdt1_2/mdt1_2 properties lustre:mgsnode=192.168.204.138@tcp lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=101 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:MDT0000 Permanent disk data: Target: lustre:OST07c6 Index: 1990 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 mgsnode=192.168.204.138@tcp:192.168.204.138@tcp autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1_2/ost1_2 properties lustre:sys.timeout=20 lustre:mgsnode=192.168.204.138@tcp:192.168.204.138@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=1990 lustre:fsname=lustre lustre:svname=lustre:OST07c6 Starting fs2mds: -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1_2/mdt1_2 Started lustre-MDT0000 Starting fs2ost: -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-lustre-OST07c6-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST07c6 mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre Creating new pool oleg438-server: Pool lustre.qpool1 created Adding targets to pool oleg438-server: OST lustre-OST07c6_UUID added to pool lustre.qpool1 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.586831 s, 34.0 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0312547 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.660273 s, 30.2 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0380718 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0464903 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.655046 s, 30.4 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0391404 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0485381 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.613411 s, 32.5 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0371186 s, 0.0 kB/s Destroy the created pools: qpool1 lustre.qpool1 oleg438-server: OST lustre-OST07c6_UUID removed from pool lustre.qpool1 oleg438-server: Pool lustre.qpool1 destroyed umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-fs2ost (opts:-f) on oleg438-server Stopping /mnt/lustre-fs2mds (opts:-f) on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. PASS 33c (101s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg438-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 34a: umount with opened file should be fail ========================================================== 18:06:11 (1715205971) start mds service on oleg438-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre multiop /mnt/lustre/file vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.20696 manual umount lustre on /mnt/lustre.... umount: /mnt/lustre: target is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 PASS 34a (38s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg438-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 34b: force umount with failed mds should be normal ========================================================== 18:06:51 (1715206011) start mds service on oleg438-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server manual umount lustre on /mnt/lustre.... umount lustre on /mnt/lustre..... stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 PASS 34b (52s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg438-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 34c: force umount with failed ost should be normal ========================================================== 18:07:45 (1715206065) start mds service on oleg438-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server manual umount lustre on /mnt/lustre.... umount lustre on /mnt/lustre..... stop ost1 service on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 PASS 34c (54s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg438-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 35a: Reconnect to the last active server first ========================================================== 18:08:40 (1715206120) start mds service on oleg438-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre debug=ha Set up a fake failnode for the MDS Wait for RECONNECT_INTERVAL seconds (10s) conf-sanity.sh test_35a 2024-05-0818h09m14s Stopping the MDT: lustre-MDT0000 stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server Restarting the MDT: lustre-MDT0000 start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 Wait for df (30354) ... done debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout Debug log: 66 lines, 66 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. oleg438-server: oleg438-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg438-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg438-client: oleg438-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg438-server: oleg438-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg438-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg438-client: oleg438-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg438-server: oleg438-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg438-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg438-client: oleg438-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: oleg438-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg438-server: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg438-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server PASS 35a (103s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 18:10:24 (1715206224) start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-05-0818h10m39s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800ad085000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-08 18:11:22.000000000 -0400 Modify: 2024-05-08 18:11:22.000000000 -0400 Change: 2024-05-08 18:11:22.000000000 -0400 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 48 lines, 48 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. oleg438-server: oleg438-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg438-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg438-client: oleg438-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg438-server: oleg438-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg438-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg438-client: oleg438-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg438-server: oleg438-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg438-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg438-client: oleg438-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: oleg438-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg438-server: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg438-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server PASS 35b (111s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 18:12:16 (1715206336) SKIP: conf-sanity test_36 remote OST SKIP 36 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 18:12:18 (1715206338) SKIP: conf-sanity test_37 ldiskfs only test SKIP 37 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:12:20 (1715206340) start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server 000000 2 000008 oleg438-server: 8+0 records in oleg438-server: 8+0 records out oleg438-server: 4096 bytes (4.1 kB) copied, 0.000674357 s, 6.1 MB/s start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 PASS 38 (58s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg438-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ========================================================== 18:13:20 (1715206400) start mds service on oleg438-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug +malloc all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug +malloc all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 PASS 39 (37s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg438-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 40: race during service thread startup ========================================================== 18:13:59 (1715206439) start ost1 service on oleg438-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid fail_loc=0x80000706 start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid umount lustre on /mnt/lustre..... stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 PASS 40 (108s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg438-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 oleg438-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg438-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================================== 18:15:49 (1715206549) start mds service on oleg438-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov -o nosvc -n lustre-mdt1/mdt1 /mnt/lustre-mds1 nomtab: 1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 start mds service on oleg438-server Starting mds1: -o localrecov -o nomgs,force lustre-mdt1/mdt1 /mnt/lustre-mds1 force: 1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre blah blah umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server PASS 41a (35s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:16:26 (1715206586) Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:) Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:-f) oleg438-server: oleg438-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg438-server Starting mds1: -o localrecov -o nosvc -n lustre-mdt1/mdt1 /mnt/lustre-mds1 nomtab: 1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Commit the device label on lustre-mdt1/mdt1 Started lustre:MDT0000 start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg438-server Starting mds1: -o localrecov -o nomgs,force lustre-mdt1/mdt1 /mnt/lustre-mds1 force: 1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre blah blah umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:-f)