-----============= acceptance-small: conf-sanity ============----- Mon Mar 16 09:37:20 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg350-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 55 63 67 32newtarball skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg350-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:39:04 (1773668344) SKIP: conf-sanity test_33a mixed loopback and real device not working SKIP 33a (2s) == conf-sanity test 33b: Drop cancel during umount ======= 09:39:05 (1773668345) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770264 s, 13.6 MB/s fail_loc=0x80000304 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 PASS 33b (56s) == conf-sanity test 33c: Mount ost with a large index number ========================================================== 09:40:01 (1773668401) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity formatting backing filesystem ldiskfs on /dev/loop0 target name lustre:MDT0000 kilobytes 200000 options -J size=8 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -J size=8 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/loop0 200000k pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Permanent disk data: Target: lustre:OST07c6 Index: 1990 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp:192.168.203.150@tcp sys.timeout=20 formatting backing filesystem ldiskfs on /dev/loop0 target name lustre:OST07c6 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST07c6 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/loop0 200000k pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Start fs2mds: mount -t lustre -o localrecov /dev/mapper/fs2mds_flakey /mnt/lustre-fs2mds oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /tmp/lustre-mdt1_2 Started lustre-MDT0000 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1837: OST_INDICES: bad array subscript pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Start fs2ost: mount -t lustre -o localrecov /dev/mapper/fs2ost_flakey /mnt/lustre-fs2ost seq.cli-lustre-OST07c6-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST07c6 /mnt/lustre-fs2ost: 83.1 MiB (87142400 bytes) trimmed mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Creating new pool qpool1 oleg350-server: Pool lustre.qpool1 created Adding targets to pool oleg350-server: OST lustre-OST07c6_UUID added to pool lustre.qpool1 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.610095 s, 32.7 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.0284092 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.495059 s, 40.2 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.0282964 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.497383 s, 40.1 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.0257625 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.0251844 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.0207431 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.0279195 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.433862 s, 45.9 MB/s Destroy the created pools: qpool1 lustre.qpool1 oleg350-server: OST lustre-OST07c6_UUID removed from pool lustre.qpool1 oleg350-server: Pool lustre.qpool1 destroyed umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-fs2ost (opts:-f) on oleg350-server Stopping /mnt/lustre-fs2mds (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. PASS 33c (135s) == conf-sanity test 33d: Don't panic when enable project quota ========================================================== 09:42:16 (1773668536) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre lfs setquota: warning: inode hardlimit '1024' smaller than minimum qunit size See 'lfs help setquota' or Lustre manual for details osd-ldiskfs.lustre-MDT0000.quota_slave.enabled=p osd-ldiskfs.lustre-MDT0001.quota_slave.enabled=p osd-ldiskfs.lustre-OST0000.quota_slave.enabled=p umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. PASS 33d (131s) == conf-sanity test 34a: umount with opened file should be fail ========================================================== 09:44:27 (1773668667) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre multiop /mnt/lustre/file vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.24724 manual umount lustre on /mnt/lustre.... umount: /mnt/lustre: target is busy. umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 34a (107s) == conf-sanity test 34b: force umount with failed mds should be normal ========================================================== 09:46:14 (1773668774) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server manual umount lustre on /mnt/lustre.... umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server stop mds service on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 34b (106s) == conf-sanity test 34c: force umount with failed ost should be normal ========================================================== 09:48:00 (1773668880) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server manual umount lustre on /mnt/lustre.... umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 34c (108s) == conf-sanity test 35a: Reconnect to the last active server first ========================================================== 09:49:49 (1773668989) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre debug=ha Set up a fake failnode for the MDS Wait for RECONNECT_INTERVAL seconds (10s) conf-sanity.sh test_35a 2026-03-16 9h50m59s Stopping the MDT: lustre-MDT0000 stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server Restarting the MDT: lustre-MDT0000 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 Wait for df (35965) ... done debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout Debug log: 105 lines, 105 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 35a (130s) == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 09:51:58 (1773669118) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2026-03-16 9h52m51s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff9cc4c5287800.stats=clear mdc.lustre-MDT0001-mdc-ffff9cc4c5287800.stats=clear Creating a test file and stat it File: /mnt/lustre/d35b.conf-sanity/f35b.conf-sanity Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115473707958274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 09:53:41.000000000 -0400 Modify: 2026-03-16 09:53:41.000000000 -0400 Change: 2026-03-16 09:53:41.000000000 -0400 Birth: 2026-03-16 09:53:40.000000000 -0400 Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 51 lines, 51 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x142 (OST update writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x142 (OST update writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 35b (176s) == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 09:54:54 (1773669294) SKIP: conf-sanity test_36 remote OST SKIP 36 (3s) == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 09:54:57 (1773669297) MDS : /dev/mapper/mds1_flakey SYMLINK : /tmp/sym_mdt.img mount symlink device - /tmp/sym_mdt.img /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' mount_op=arg[0] = /sbin/mount.lustre arg[1] = -v arg[2] = -o arg[3] = rw,localrecov arg[4] = /dev/mapper/mds1_flakey arg[5] = /mnt/lustre-mds1 source = /dev/mapper/mds1_flakey (/dev/mapper/mds1_flakey), target = /mnt/lustre-mds1 options(13/4096) = rw,localrecov checking for existing Lustre data: found mounting device /dev/mapper/mds1_flakey at /mnt/lustre-mds1, flags=0x1000000 options=user_xattr,errors=remount-ro,localrecov,osd=osd-ldiskfs,mgs,writeconf,param=sys.timeout=20,param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity,svname=lustre-MDT0000,device=/dev/mapper/mds1_flakey PASS 37 (33s) == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:55:30 (1773669330) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=300 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=258 stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server delete lov_objid file on MDS 000000 299 000008 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=290 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server 000000 2 000008 oleg350-server: 8+0 records in oleg350-server: 8+0 records out oleg350-server: 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000565239 s, 7.2 MB/s start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=322 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server stop mds service on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 38 (198s) == conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ========================================================== 09:58:49 (1773669529) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug +malloc all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug +malloc all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug +malloc all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 39 (85s) == conf-sanity test 40: race during service thread startup ========================================================== 10:00:14 (1773669614) start ost1 service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid fail_loc=0x80000706 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 40 (102s) == conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================================== 10:01:56 (1773669716) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov -o nosvc -n /dev/mapper/mds1_flakey /mnt/lustre-mds1 nomtab: 1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all Start /dev/mapper/mds1_flakey without service Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov -o nomgs,force /dev/mapper/mds1_flakey /mnt/lustre-mds1 force: 1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre blah blah umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server PASS 41a (62s) == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:02:58 (1773669778) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov -o nosvc -n /dev/mapper/mds1_flakey /mnt/lustre-mds1 nomtab: 1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all Start /dev/mapper/mds1_flakey without service Commit the device label on /dev/mapper/mds1_flakey Started lustre:MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov -o nomgs,force /dev/mapper/mds1_flakey /mnt/lustre-mds1 force: 1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre blah blah umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server PASS 41b (98s) == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:04:36 (1773669876) umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server stop mds service on oleg350-server stop mds service on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' fail_loc=0x80000716 oleg350-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Operation already in progress oleg350-server: The target service is already running. (/dev/mapper/mds1_flakey) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 114 fail_loc=0x0 1st MDT start failed with 114 2nd MDT start succeed start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-server: oleg350-server.virtnet: executing lsmod fail_loc=0x80000716 oleg350-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Operation already in progress oleg350-server: The target service is already running. (/dev/mapper/ost1_flakey) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 114 fail_loc=0x0 1st OST start succeed 2nd OST start failed with 114 stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 41c (156s) == conf-sanity test 42: allow client/server mount/unmount with invalid config param ========================================================== 10:07:12 (1773670032) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 PASS 42 (101s) == conf-sanity test 43a: check root_squash and nosquash_nids ========================================================== 10:08:53 (1773670133) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Setting lustre.mdt.root_squash from 0:0 to 0:0 Setting lustre.mdt.nosquash_nids from NONE to NONE Setting lustre.mdt.root_squash from 0:0 to 500:500 Waiting 90s for '500:500' Updated after 7s: want '500:500' got '500:500' /mnt/lustre/f43a.conf-sanity-userfile: owner uid 500 (-rw-------): root read permission is granted - ok /mnt/lustre/f43a.conf-sanity-userfile: owner uid 500 (-rw-------): root write permission is granted - ok /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root read permission is denied - ok /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root write permission is denied - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root unlink permission is denied - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root create permission is denied - ok /mnt/lustre/f43a.conf-sanity-user1file: owner uid 501 (-rw-------): root read permission is denied - ok /mnt/lustre/f43a.conf-sanity-user1file: owner uid 501 (-rw-------): root write permission is denied - ok /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4308: 79903 Terminated runas -u $ID1 tail -f $DIR/$tfile-user1file > /dev/null 2>&1 Setting lustre-MDTall.mdt.nosquash_nids from NONE to 2@gni 0@lo 192.168.203.50@tcp 192.168.0.[2,10]@tcp Waiting 90s for '2@gni 0@lo 192.168.203.50@tcp 192.168.0.[2,10]@tcp' Updated after 5s: want '2@gni 0@lo 192.168.203.50@tcp 192.168.0.[2,10]@tcp' got '2@gni 0@lo 192.168.203.50@tcp 192.168.0.[2,10]@tcp' /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root read permission is granted - ok /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root write permission is granted - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root unlink permission is granted - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root create permission is granted - ok Setting lustre-MDTall.mdt.nosquash_nids from 2@gni 0@lo 192.168.203.50@tcp 192.168.0.[2,10]@tcp to 192.168.203.50/24@tcp Waiting 90s for '192.168.203.50/24@tcp' Updated after 5s: want '192.168.203.50/24@tcp' got '192.168.203.50/24@tcp' Waiting 90s for '192.168.203.50/24@tcp' /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root read permission is granted - ok /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root write permission is granted - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root unlink permission is granted - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root create permission is granted - ok umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 43a (93s) == conf-sanity test 43b: parse nosquash_nids with commas in expr_list ========================================================== 10:10:26 (1773670226) SKIP: conf-sanity test_43b mixed loopback and real device not working SKIP 43b (2s) == conf-sanity test 44: mounted client proc entry exists ========================================================== 10:10:28 (1773670228) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 44 (68s) SKIP: conf-sanity test_45 skipping SLOW test 45 == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 10:11:36 (1773670296) Testing with 2 OSTs Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cc4c45c1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cc4c45c1000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 34 0x22 0x280000401 1 2 0x2 0x2c0000401 File: /mnt/lustre/widestripe Size: 3 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 10:13:34.000000000 -0400 Modify: 2026-03-16 10:13:34.000000000 -0400 Change: 2026-03-16 10:13:35.000000000 -0400 Birth: 2026-03-16 10:13:34.000000000 -0400 umount lustre on /mnt/lustre2..... Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server stop ost1 service on oleg350-server stop mds service on oleg350-server stop mds service on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 PASS 46a (157s) == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 10:14:13 (1773670453) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success ldlm.namespaces.lustre-MDT0000-mdc-ffff9cc4c8df1800.lru_size=100 ldlm.namespaces.lustre-MDT0001-mdc-ffff9cc4c8df1800.lru_size=100 ldlm.namespaces.lustre-OST0000-osc-ffff9cc4c8df1800.lru_size=100 ldlm.namespaces.lustre-MDT0000-mdc-ffff9cc4c8df1800.lru_size=100 ldlm.namespaces.lustre-MDT0001-mdc-ffff9cc4c8df1800.lru_size=100 Failing ost1 on oleg350-server Stopping /mnt/lustre-ost1 (opts:) on oleg350-server 10:15:02 (1773670502) shut down facet: ost1 facet_host: oleg350-server facet_failover_host: oleg350-server Failover ost1 to oleg350-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 10:15:17 (1773670517) targets are mounted 10:15:17 (1773670517) facet_failover done Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 10:15:18 (1773670518) shut down facet: mds1 facet_host: oleg350-server facet_failover_host: oleg350-server Failover mds1 to oleg350-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 10:15:32 (1773670532) targets are mounted 10:15:32 (1773670532) facet_failover done ldlm.namespaces.lustre-MDT0000-mdc-ffff9cc4c8df1800.lru_size=100 ldlm.namespaces.lustre-MDT0001-mdc-ffff9cc4c8df1800.lru_size=100 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 47a (105s) == conf-sanity test 47b: client restart does not lose lru_resize ========================================================== 10:15:58 (1773670558) Failing mds1 on oleg350-server 10:16:00 (1773670560) shut down facet: mds1 facet_host: oleg350-server facet_failover_host: oleg350-server Failover mds1 to oleg350-server mount facets: mds1 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 10:16:25 (1773670585) targets are mounted 10:16:25 (1773670585) facet_failover done pdsh@oleg350-client: oleg350-client: ssh exited with exit code 95 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg350-client: oleg350-client: ssh exited with exit code 95 Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success ln: failed to create symbolic link '/home/green/git/lustre-release/lustre/utils/lctl': File exists SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl SKIP 47b (77s) == conf-sanity test 48: too many acls on file ============ 10:17:15 (1773670635) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg350-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab 192.168.203.150@tcp:/lustre is already mounted on /mnt/lustre setup single mount lustre success debug=0 debug=0 /mnt/lustre stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/widestripe lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4 0x4 0x280000401 It is expected to hold at least 4500 ACL entries File: /mnt/lustre/widestripe Size: 3 Blocks: 8 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289279491 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 10:17:21.000000000 -0400 Modify: 2026-03-16 10:17:21.000000000 -0400 Change: 2026-03-16 10:18:02.000000000 -0400 Birth: 2026-03-16 10:17:20.000000000 -0400 getfacl: Removing leading '/' from absolute path names Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg350-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server PASS 48 (147s) == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:19:42 (1773670782) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success check ldlm_timout... umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server PASS 49a (60s) == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:20:42 (1773670842) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 49b (70s) == conf-sanity test 50a: lazystatfs all servers available ========================================================== 10:21:52 (1773670912) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre llite.lustre-ffff9cc4c4170000.lazystatfs=1 multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.8173 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1616 80976 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1492 81100 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1528 126688 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1528 126688 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4706: kill: (123039) - No such process umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 PASS 50a (74s) == conf-sanity test 50b: lazystatfs all servers down ===== 10:23:06 (1773670986) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre llite.lustre-ffff9cc4c658e000.lazystatfs=1 stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server oleg350-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9cc4c658e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cc4c658e000.ost_server_uuid in DISCONN state after 0 sec 0 UP mgc MGC192.168.203.150@tcp aed47d24-a649-4418-9d4d-ce072f1d15bb 5 1 UP lov lustre-clilov-ffff9cc4c658e000 7abfc49d-d4a2-4531-a4fa-c3b77f4ae617 4 2 UP lmv lustre-clilmv-ffff9cc4c658e000 7abfc49d-d4a2-4531-a4fa-c3b77f4ae617 5 3 UP mdc lustre-MDT0000-mdc-ffff9cc4c658e000 7abfc49d-d4a2-4531-a4fa-c3b77f4ae617 5 4 UP mdc lustre-MDT0001-mdc-ffff9cc4c658e000 7abfc49d-d4a2-4531-a4fa-c3b77f4ae617 5 5 UP osc lustre-OST0000-osc-ffff9cc4c658e000 7abfc49d-d4a2-4531-a4fa-c3b77f4ae617 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.8173 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1616 80976 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1492 81100 2% /mnt/lustre[MDT:1] filesystem_summary: 190496 3108 162076 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4706: kill: (127110) - No such process umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server PASS 50b (73s) == conf-sanity test 50c: lazystatfs one server down ====== 10:24:19 (1773671059) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg350-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre llite.lustre-ffff9cc4c79e9000.lazystatfs=1 stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server oleg350-server: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.8173 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1712 80880 3% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1524 81068 2% /mnt/lustre[MDT:1] lustre-OST0001_UUID 142216 1544 126672 2% /mnt/lustre[OST:1] filesystem_summary: 142216 1544 126672 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4706: kill: (129756) - No such process umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 PASS 50c (66s) == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:25:25 (1773671125) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg350-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre llite.lustre-ffff9cc4c846d800.lazystatfs=1 stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.8173 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1712 80880 3% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1520 81072 2% /mnt/lustre[MDT:1] lustre-OST0001_UUID 142216 1544 126672 2% /mnt/lustre[OST:1] filesystem_summary: 142216 1544 126672 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4706: kill: (132504) - No such process umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 sys.ldlm_timeout=19 PASS 50d (73s) == conf-sanity test 50e: normal statfs all servers down == 10:26:38 (1773671198) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server oleg350-server: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre llite.lustre-ffff9cc4d0982000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.8173 start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server PASS 50e (147s) == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:29:05 (1773671345) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg350-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server oleg350-server: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre llite.lustre-ffff9cc4c63d2800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.8173 start ost2 service on oleg350-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 PASS 50f (164s) == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:31:49 (1773671509) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre start ost2 service on oleg350-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cc4c6256800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cc4c6256800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.150@tcp:/lustre 142216 1528 126688 2% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 PASS 50g (86s) == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:33:15 (1773671595) checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x142 (OST update writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 osc.active=0 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg350-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Setting lustre-OST0000.osc.active from 0 to 1 Waiting 90s for '1' Updated after 3s: want '1' got '1' create a file after OST1 is activated 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.601673 s, 17.4 MB/s umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 50h (84s) == conf-sanity test 50i: activate deactivated MDT ======== 10:34:39 (1773671679) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x41 (MDT update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mdc.active=0 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg350-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Reader error: 'No devices found' at 0 No device found for name MGS: Invalid argument This command must be run on the MGS. error: conf_param: No such device Setting lustre-MDT0001.mdc.active from 0 to 1 Waiting 90s for '1' oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 oleg350-server: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 oleg350-server: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 4 sec total: 1 open/close in 0.02 seconds: 50.61 ops/second Setting lustre-MDT0001.mdc.active from 1 to 0 Waiting 90s for '0' Updated after 9s: want '0' got '0' osp.lustre-MDT0001-osp-MDT0000.active updated after 0 sec (got 0) lfs mkdir: dirstripe error on '/mnt/lustre/d50i.conf-sanity/2': No such device lfs setdirstripe: cannot create dir '/mnt/lustre/d50i.conf-sanity/2': No such device umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop ost1 service on oleg350-server PASS 50i (77s) == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:35:56 (1773671756) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg350-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 51 (109s) == conf-sanity test 52: check recovering objects from lost+found ========================================================== 10:37:45 (1773671865) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre ........ backup files to /tmp/d52.conf-sanity tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets getfattr: Removing leading '/' from absolute path names umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server mount ost1 as ldiskfs backup objects to /tmp/conf52/objects oleg350-server: tar: Removing leading `/' from member names oleg350-server: tar: Removing leading `/' from hard link targets oleg350-server: getfattr: Removing leading '/' from absolute path names start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets getfattr: Removing leading '/' from absolute path names umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 PASS 52 (157s) == conf-sanity test 53a: check OSS thread count params === 10:40:22 (1773672022) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success checking (($tstarted && $tmin && $tmax)) (((4 && 3 && 116)))... checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((4 >= 3 && 4 <= 116 )))... ost.OSS.ost.threads_min=4 ost.OSS.ost.threads_max=115 checking (($tmin2 == ($tmin + $nthrs) && $tmax2 == ($tmax - $nthrs))) (((4 == (3 + 1) && 115 == (116 - 1))))... ost.OSS.ost.threads_max=4 checking (($tmin2 == $tmin && $tmax2 == $tmin)) (((4 == 4 && 4 == 4)))... oleg350-server: error: set_param: setting /sys/fs/lustre/ost/OSS/ost/threads_max=3: Numerical result out of range oleg350-server: error: set_param: setting 'ost/OSS/ost/threads_max'='3': Numerical result out of range pdsh@oleg350-client: oleg350-server: ssh exited with exit code 34 checking (($tmin2 <= $tmax2)) (((4 <= 4)))... umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. setmodopts: MODOPTS_OST= oss_num_threads=16 start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ost/ost options: 'oss_num_threads=16' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg350-server: ost/ost options: 'oss_num_threads=16' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((4 >= 3 && 4 <= 116 )))... umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 PASS 53a (284s) == conf-sanity test 53b: check MDS thread count params === 10:45:06 (1773672306) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success checking (($tstarted && $tmin && $tmax)) (((4 && 3 && 116)))... checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((4 >= 3 && 4 <= 116 )))... mds.MDS.mdt..threads_min=4 mds.MDS.mdt..threads_max=115 checking (($tmin2 == ($tmin + $nthrs) && $tmax2 == ($tmax - $nthrs))) (((4 == (3 + 1) && 115 == (116 - 1))))... mds.MDS.mdt..threads_max=4 checking (($tmin2 == $tmin && $tmax2 == $tmin)) (((4 == 4 && 4 == 4)))... oleg350-server: error: set_param: setting /sys/fs/lustre/mds/MDS/mdt//threads_max=3: Numerical result out of range oleg350-server: error: set_param: setting 'mds/MDS/mdt//threads_max'='3': Numerical result out of range pdsh@oleg350-client: oleg350-server: ssh exited with exit code 34 checking (($tmin2 <= $tmax2)) (((4 <= 4)))... umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. setmodopts: MODOPTS_MDT=mdt_enable_flr_ec=1 mds_num_threads=16 start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1 mds_num_threads=16' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1 mds_num_threads=16' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success setmodopts: MODOPTS_MDT=mdt_enable_flr_ec=1 checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((16 >= 16 && 16 <= 16 )))... umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 53b (293s) == conf-sanity test 54a: test llverdev and partial verify of device ========================================================== 10:49:59 (1773672599) oleg350-server: oleg350-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p oleg350-server: llverdev: /dev/mapper/ost1_flakey is 4294967296 bytes (4 GB) in size oleg350-server: timestamp: 1773672604 chunksize: 1048576 size: 4294967296 oleg350-server: write offset: 4194304kB inf MB/s oleg350-server: read offset: 4194304kB inf MB/s oleg350-server: llverdev: data verified successfully Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server PASS 54a (184s) == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:53:03 (1773672783) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Timestamp: 1773672845 dirs: 2, fs blocks: 35554 write_done: /mnt/lustre/llverfs_dir00001/file000, current: 2.78577 MB/s, overall: 2.78577 MB/s, ETA: 0:00:00 read_done: /mnt/lustre/llverfs_dir00001/file000, current: 56.9719 MB/s, overall: 56.9719 MB/s, ETA: 0:00:00 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 54b (114s) SKIP: conf-sanity test_55 skipping excluded test 55 == conf-sanity test 56a: check big OST indexes and out-of-index-order start ========================================================== 10:54:59 (1773672899) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey Permanent disk data: Target: lustre:OST2710 Index: 10000 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST2710 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST2710 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Permanent disk data: Target: lustre:OST03e8 Index: 1000 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost2_flakey target name lustre:OST03e8 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST03e8 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost2_flakey 200000k start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST2710-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST2710 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg350-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST03e8-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST03e8 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre ok OBDS: 1000: lustre-OST03e8_UUID ACTIVE 10000: lustre-OST2710_UUID ACTIVE 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00760885 s, 538 kB/s oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:) Stopping client oleg350-client.virtnet /mnt/lustre opts: Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server PASS 56a (392s) == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:01:32 (1773673292) SKIP: conf-sanity test_56b needs >= 3 MDTs SKIP 56b (3s) == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:01:35 (1773673295) oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x142 (OST update writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 failover.node=192.168.203.150@tcp start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg350-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Cannot assign requested address pdsh@oleg350-client: oleg350-server: ssh exited with exit code 99 oleg350-server: error: set_param: param_path 'seq/cli-lustre': No such file or directory oleg350-server: error: set_param: setting 'seq/cli-lustre'='OST0000-super.width=65536': No such file or directory pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 Start of /dev/mapper/ost1_flakey on ost1 failed 99 umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 57a (117s) == conf-sanity test 57b: initial registration from servicenode should not fail ========================================================== 11:03:32 (1773673412) oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 failover.node=192.168.203.150@tcp Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 failover.node=192.168.203.150@tcp checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x142 (OST update writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x142 (OST update writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 failover.node=192.168.203.150@tcp Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1142 (OST update writeconf no_primnode ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 failover.node=192.168.203.150@tcp:192.168.203.150@tcp start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 57b (194s) == conf-sanity test 58: missing llog files must not prevent MDT from mounting ========================================================== 11:06:46 (1773673606) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre total: 100 open/close in 2.22 seconds: 45.09 ops/second - unlinked 0 (time 1773673693 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 58 (210s) == conf-sanity test 59: writeconf mount option =========== 11:10:16 (1773673816) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy ln: failed to create symbolic link '/sbin/.libs': Read-only file system oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server original ost count: 1 (expect > 0) oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid after mdt writeconf count: 0 (expect 0) OST start without writeconf should fail: oleg350-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: No such file or directory oleg350-server: Is the MGS specification correct? oleg350-server: Is the filesystem name correct? oleg350-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg350-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 OST start with writeconf should succeed: oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid after ost writeconf count: 1 (expect 1) oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid after ost2 writeconf count: 2 (expect 2) oleg350-server: oleg350-server.virtnet: executing unload_modules_local checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1002 (OST no_primnode ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 failover.node=192.168.203.150@tcp:192.168.203.150@tcp Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1102 (OST writeconf no_primnode ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 failover.node=192.168.203.150@tcp:192.168.203.150@tcp checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 59 (255s) == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 11:14:31 (1773674071) Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 200000 options -I 1024 -i 2560 -q -O ^uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -I 1024 -i 2560 -q -O ^uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k Permanent disk data: Target: lustre:MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x61 (MDT first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds2_flakey target name lustre:MDT0001 kilobytes 200000 options -I 1024 -i 2560 -q -O ^uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0001 -I 1024 -i 2560 -q -O ^uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/mapper/mds2_flakey 200000k oleg350-server: dumpe2fs 1.47.3-wc2 (11-Nov-2025) stop mds service on oleg350-server stop mds service on oleg350-server Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server PASS 60a (196s) == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:17:47 (1773674267) oleg350-server: dumpe2fs 1.47.3-wc2 (11-Nov-2025) Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg ea_inode dirdata large_dir sparse_super large_file huge_file uninit_bg dir_nlink quota project Journal features: journal_incompat_revoke PASS 60b (8s) == conf-sanity test 61a: large xattr ===================== 11:17:55 (1773674275) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity grow value of trusted.big on /mnt/lustre/f61a.conf-sanity check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 11:19:12 (1773674352) shut down facet: mds1 facet_host: oleg350-server facet_failover_host: oleg350-server Failover mds1 to oleg350-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 11:19:46 (1773674386) targets are mounted 11:19:46 (1773674386) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 11:20:02 (1773674402) shut down facet: mds1 facet_host: oleg350-server facet_failover_host: oleg350-server Failover mds1 to oleg350-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 11:20:33 (1773674433) targets are mounted 11:20:33 (1773674433) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:) Stopping client oleg350-client.virtnet /mnt/lustre opts: Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8 oleg350-server: e2fsck 1.47.3-wc2 (11-Nov-2025) oleg350-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2], used inodes 274/79992 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 82 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 84 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 88 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 88 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 89 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 90 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 91 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 92 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 93 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 94 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 94 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 96 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 97 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 98 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 104 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 106 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 108 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 108 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 109 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 109 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 110 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 110 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 122 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 124 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 125 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 125 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 126 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 131 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 131 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 132 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 132 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 133 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 133 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 136 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 148 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 148 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 149 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 149 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 150 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 151 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 152 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 153 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 153 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 157 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 158 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 159 badness 0 to 2 for 10084 [Thread 0] group 0 finished [Thread 0] e2fsck_pass1_run:2316: increase inode 26683 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26683 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26710 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26710 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26711 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26711 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26712 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26712 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26713 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26713 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26714 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26714 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26715 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2316: increase inode 26716 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26716 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26717 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26717 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26718 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26718 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26719 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26719 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26720 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26720 badness 1 to 3 for 10084 [Thread 0] group 1 finished [Thread 0] group 2 finished [Thread 0] Pass 1: Memory used: 280k/0k (155k/126k), time: 0.02/ 0.00/ 0.00 [Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 47.65MB/s [Thread 0] Scanned group range [0, 2], used inodes 274/276 Pass 2: Checking directory structure Pass 2: Memory used: 280k/0k (107k/174k), time: 0.02/ 0.00/ 0.00 Pass 2: I/O read: 1MB, write: 0MB, rate: 56.95MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 280k/0k (107k/174k), time: 0.06/ 0.00/ 0.01 Pass 3A: Memory used: 280k/0k (107k/174k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 280k/0k (105k/176k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 6535.95MB/s Pass 4: Checking reference counts Pass 4: Memory used: 280k/0k (78k/203k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 5: Checking group summary information Pass 5: Memory used: 388k/0k (77k/312k), time: 0.01/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 105.43MB/s 274 inodes used (0.34%, out of 79992) 3 non-contiguous files (1.1%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 202 24540 blocks used (49.08%, out of 50000) 0 bad blocks 1 large file 147 regular files 117 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 264 files Memory used: 388k/0k (81k/308k), time: 0.09/ 0.01/ 0.01 I/O read: 1MB, write: 1MB, rate: 10.97MB/s e2fsck -d -v -t -t -f -y /dev/mapper/mds2_flakey -m8 oleg350-server: e2fsck 1.47.3-wc2 (11-Nov-2025) oleg350-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2], used inodes 254/79992 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 85 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 85 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 86 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 87 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 88 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 91 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 91 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 92 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 92 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 93 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 94 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 94 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 96 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 97 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 98 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 99 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 112 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 114 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 114 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 115 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 115 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 116 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 117 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 119 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 122 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 124 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 125 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 125 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 126 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 131 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 132 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 132 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 133 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 133 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 136 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 137 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 0 to 2 for 10084 [Thread 0] group 0 finished [Thread 0] group 1 finished [Thread 0] group 2 finished [Thread 0] Pass 1: Memory used: 280k/0k (155k/126k), time: 0.01/ 0.00/ 0.00 [Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 130.31MB/s [Thread 0] Scanned group range [0, 2], used inodes 254/256 Pass 2: Checking directory structure Pass 2: Memory used: 280k/0k (107k/174k), time: 0.02/ 0.00/ 0.00 Pass 2: I/O read: 1MB, write: 0MB, rate: 44.12MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 280k/0k (107k/174k), time: 0.05/ 0.00/ 0.00 Pass 3A: Memory used: 280k/0k (108k/173k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 280k/0k (106k/175k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 700.77MB/s Pass 4: Checking reference counts Pass 4: Memory used: 280k/0k (78k/203k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 5: Checking group summary information Pass 5: Memory used: 388k/0k (77k/312k), time: 0.01/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 77.99MB/s 254 inodes used (0.32%, out of 79992) 2 non-contiguous files (0.8%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 197 24509 blocks used (49.02%, out of 50000) 0 bad blocks 1 large file 135 regular files 109 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 244 files Memory used: 388k/0k (81k/308k), time: 0.08/ 0.01/ 0.01 I/O read: 1MB, write: 1MB, rate: 12.72MB/s start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 61a (292s) == conf-sanity test 61b: large xattr ===================== 11:22:47 (1773674567) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:) Stopping client oleg350-client.virtnet /mnt/lustre opts: Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server oleg350-server: debugfs 1.47.3-wc2 (11-Nov-2025) large ea <163> oleg350-server: debugfs 1.47.3-wc2 (11-Nov-2025) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Started LFSCK on the device lustre-MDT0000: scrub namespace Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:) Stopping client oleg350-client.virtnet /mnt/lustre opts: Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8 oleg350-server: e2fsck 1.47.3-wc2 (11-Nov-2025) oleg350-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2], used inodes 276/79992 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 82 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 84 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 88 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 88 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 89 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 90 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 91 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 92 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 93 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 94 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 94 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 96 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 97 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 98 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 104 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 106 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 108 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 108 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 109 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 109 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 110 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 110 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 122 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 124 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 125 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 125 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 126 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 131 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 131 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 132 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 132 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 133 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 133 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 136 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 148 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 148 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 149 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 149 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 150 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 151 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 152 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 153 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 153 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 157 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 158 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 159 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2660: increase inode 163 badness 0 to 1 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 163 badness 1 to 3 for 10084 [Thread 0] group 0 finished [Thread 0] e2fsck_pass1_run:2316: increase inode 26683 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26683 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26710 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26710 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26711 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26711 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26712 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26712 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26713 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26713 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26714 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26714 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26715 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2316: increase inode 26716 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26716 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26717 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26717 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26718 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26718 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26719 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26719 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26720 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26720 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26721 badness 0 to 1 for 1005c [Thread 0] group 1 finished [Thread 0] group 2 finished [Thread 0] Pass 1: Memory used: 280k/0k (155k/126k), time: 0.02/ 0.00/ 0.00 [Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 61.56MB/s [Thread 0] Scanned group range [0, 2], used inodes 276/277 Pass 2: Checking directory structure check_dir_block:1608: increase inode 11 badness 0 to 1 for 20055 Entry '<163>' in /lost+found (11) references EA inopdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 de 163. Clear? yes Pass 2: Memory used: 280k/0k (107k/174k), time: 0.02/ 0.00/ 0.00 Pass 2: I/O read: 1MB, write: 1MB, rate: 41.85MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 280k/0k (107k/174k), time: 0.06/ 0.01/ 0.00 Pass 3A: Memory used: 280k/0k (108k/173k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 280k/0k (106k/175k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 8403.36MB/s Pass 4: Checking reference counts Pass 4: Memory used: 280k/0k (79k/202k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 403.71MB/s Pass 5: Checking group summary information Pass 5: Memory used: 388k/0k (78k/311k), time: 0.01/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 80.36MB/s lustre-MDT0000: ***** FILE SYSTEM WAS MODIFIED ***** 276 inodes used (0.35%, out of 79992) 3 non-contiguous files (1.1%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 203 24557 blocks used (49.11%, out of 50000) 0 bad blocks 1 large file 148 regular files 118 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 265 files Memory used: 388k/0k (81k/308k), time: 0.09/ 0.02/ 0.00 I/O read: 1MB, write: 1MB, rate: 10.66MB/s e2fsck -d -v -t -t -f -y /dev/mapper/mds2_flakey -m8 oleg350-server: e2fsck 1.47.3-wc2 (11-Nov-2025) oleg350-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2], used inodes 254/79992 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 85 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 85 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 86 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 87 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 88 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 91 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 91 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 92 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 92 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 93 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 94 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 94 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 96 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 97 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 98 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 99 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 112 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 114 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 114 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 115 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 115 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 116 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 117 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 119 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 122 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 124 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 125 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 125 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 126 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 131 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 132 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 132 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 133 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 133 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 136 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 137 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 0 to 2 for 10084 [Thread 0] group 0 finished [Thread 0] group 1 finished [Thread 0] group 2 finished [Thread 0] Pass 1: Memory used: 280k/0k (155k/126k), time: 0.01/ 0.00/ 0.00 [Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 106.85MB/s [Thread 0] Scanned group range [0, 2], used inodes 254/256 Pass 2: Checking directory structure Pass 2: Memory used: 280k/0k (107k/174k), time: 0.02/ 0.00/ 0.00 Pass 2: I/O read: 1MB, write: 0MB, rate: 46.98MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 280k/0k (107k/174k), time: 0.04/ 0.01/ 0.01 Pass 3A: Memory used: 280k/0k (108k/173k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 280k/0k (106k/175k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 5681.82MB/s Pass 4: Checking reference counts Pass 4: Memory used: 280k/0k (78k/203k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 5: Checking group summary information Pass 5: Memory used: 388k/0k (77k/312k), time: 0.01/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 88.47MB/s 254 inodes used (0.32%, out of 79992) 2 non-contiguous files (0.8%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 197 24509 blocks used (49.02%, out of 50000) 0 bad blocks 1 large file 135 regular files 109 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 244 files Memory used: 388k/0k (81k/308k), time: 0.07/ 0.01/ 0.01 I/O read: 1MB, write: 1MB, rate: 14.34MB/s start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 61b (313s) == conf-sanity test 62: start with disabled journal ====== 11:28:00 (1773674880) disable journal for mds tune2fs 1.47.3-wc2 (11-Nov-2025) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Invalid argument oleg350-server: This may have multiple causes. oleg350-server: Are the mount options correct? oleg350-server: Check the syslog for more info. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 Start of /dev/mapper/mds1_flakey on mds1 failed 22 disable journal for ost tune2fs 1.47.3-wc2 (11-Nov-2025) start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg350-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Invalid argument oleg350-server: This may have multiple causes. oleg350-server: Are the mount options correct? oleg350-server: Check the syslog for more info. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 oleg350-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg350-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 Start of /dev/mapper/ost1_flakey on ost1 failed 22 umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server stop mds service on oleg350-server stop mds service on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server PASS 62 (214s) SKIP: conf-sanity test_63 skipping excluded test 63 == conf-sanity test 64: check lfs df --lazy ============== 11:31:36 (1773675096) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg350-server Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre stop ost2 service on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server /home/green/git/lustre-release/lustre/utils/lfs df UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1664 80928 3% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1524 81068 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1528 126688 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1528 126688 2% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 64 (163s) == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 11:34:19 (1773675259) stop mds service on oleg350-server stop mds service on oleg350-server oleg350-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg350-server: mount: /mnt/lustre-brpt: unknown filesystem type 'ldiskfs'. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 32 oleg350-server: umount: /mnt/lustre-brpt: not mounted. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 32 start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server oleg350-server: debugfs 1.47.3-wc2 (11-Nov-2025) PASS 65 (94s) == conf-sanity test 66: replace nids ===================== 11:35:53 (1773675353) client=34681601 MDS=34681601 OSS=34681601 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Setting lustre-OST0000.osc.active from 1 to 0 Waiting 90s for '0' Updated after 9s: want '0' got '0' replace_nids should fail if MDS, OSTs and clients are UP oleg350-server: error: replace_nids: Operation now in progress pdsh@oleg350-client: oleg350-server: ssh exited with exit code 115 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) replace_nids should fail if MDS and OSTs are UP oleg350-server: error: replace_nids: Operation now in progress pdsh@oleg350-client: oleg350-server: ssh exited with exit code 115 stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server replace_nids should fail if MDS is UP oleg350-server: error: replace_nids: Operation now in progress pdsh@oleg350-client: oleg350-server: ssh exited with exit code 115 stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov -o nosvc /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all Start /dev/mapper/mds1_flakey without service Started lustre-MDT0000 command should accept two parameters pdsh@oleg350-client: oleg350-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] correct device name should be passed oleg350-server: error: replace_nids: No such device or address pdsh@oleg350-client: oleg350-server: ssh exited with exit code 6 wrong nids list should not destroy the system pdsh@oleg350-client: oleg350-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] pdsh@oleg350-client: oleg350-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] replace OST nid command should accept two parameters pdsh@oleg350-client: oleg350-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] wrong nids list should not destroy the system pdsh@oleg350-client: oleg350-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] set NIDs with failover replace MDS nid stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Setting lustre-OST0000.osc.active from 0 to 1 Waiting 90s for '1' Waiting 70s for '1' Updated after 28s: want '1' got '1' start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server PASS 66 (430s) SKIP: conf-sanity test_67 skipping excluded test 67 == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:43:04 (1773675784) umount lustre on /mnt/lustre..... start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid seq.ctl-lustre-MDT0000.fldb=[0x2c0000400-12884902912):0:mdt seq.srv-lustre-MDT0000.space=clear mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre seq.srv-lustre-MDT0000.space=[0x300000400 - 0x340000400]:0:mdt [0x200000bd1:0x1:0x0] umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 68 (100s) SKIP: conf-sanity test_69 skipping SLOW test 69 == conf-sanity test 70a: start MDT0, then OST, then MDT1 ========================================================== 11:44:46 (1773675886) umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server stop mds service on oleg350-server stop mds service on oleg350-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: oleg350-server: LNET unconfigure error 22: Invalid argument modules unloaded. start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 70a (108s) == conf-sanity test 70b: start OST, MDT1, MDT0 =========== 11:46:34 (1773675994) start ost1 service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 70b (150s) == conf-sanity test 70c: stop MDT0, mkdir fail, create remote dir fail ========================================================== 11:49:04 (1773676144) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server deactivate lustre-MDT0000-mdc-ffff9cc4c8b33800 mkdir: cannot create directory '/mnt/lustre/d70c.conf-sanity': Cannot send after transport endpoint shutdown lfs mkdir: cannot resolve path '/mnt/lustre/d70c.conf-sanity/remote_dir': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d70c.conf-sanity/remote_dir' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d70c.conf-sanity/remote_dir': Cannot send after transport endpoint shutdown umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) lsof: status error on /mnt/lustre: Cannot send after transport endpoint shutdown lsof 4.93.2 latest revision: https://github.com/lsof-org/lsof latest FAQ: https://github.com/lsof-org/lsof/blob/master/00FAQ latest (non-formatted) man page: https://github.com/lsof-org/lsof/blob/master/Lsof.8 usage: [-?abhKlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-E] [+|-e s] [+|-f[gG]] [-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s] [+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names] Use the ``-h'' option to get more help information. stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 PASS 70c (116s) == conf-sanity test 70d: stop MDT1, mkdir succeed, create remote dir fail ========================================================== 11:51:00 (1773676260) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server deactivate lustre-MDT0001-mdc-ffff9cc4d0987000 lfs mkdir: dirstripe error on '/mnt/lustre/d70d.conf-sanity/remote_dir': No such device lfs setdirstripe: cannot create dir '/mnt/lustre/d70d.conf-sanity/remote_dir': No such device umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 PASS 70d (125s) == conf-sanity test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 11:53:05 (1773676385) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 200000 options -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Permanent disk data: Target: lustre:MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x61 (MDT first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds2_flakey target name lustre:MDT0001 kilobytes 200000 options -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0001 -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds2_flakey 200000k start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-server: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 oleg350-server: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 oleg350-server: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 PASS 70e (201s) == conf-sanity test 71a: start MDT0 OST0, MDT1, OST1 ===== 11:56:26 (1773676586) SKIP: conf-sanity test_71a needs separate MGS/MDT SKIP 71a (3s) == conf-sanity test 71b: start MDT1, OST0, MDT0, OST1 ==== 11:56:29 (1773676589) SKIP: conf-sanity test_71b needs separate MGS/MDT SKIP 71b (2s) == conf-sanity test 71c: start OST0, OST1, MDT1, MDT0 ==== 11:56:31 (1773676591) SKIP: conf-sanity test_71c needs separate MGS/MDT SKIP 71c (3s) == conf-sanity test 71d: start OST0, MDT1, MDT0, OST1 ==== 11:56:34 (1773676594) SKIP: conf-sanity test_71d needs separate MGS/MDT SKIP 71d (2s) == conf-sanity test 71e: start OST0, MDT1, OST1, MDT0 ==== 11:56:36 (1773676596) SKIP: conf-sanity test_71e needs separate MGS/MDT SKIP 71e (2s) == conf-sanity test 72: test fast symlink with extents flag enabled ========================================================== 11:56:38 (1773676598) umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server stop mds service on oleg350-server stop mds service on oleg350-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: oleg350-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 200000 options -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k tune2fs 1.47.3-wc2 (11-Nov-2025) Permanent disk data: Target: lustre:MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x61 (MDT first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds2_flakey target name lustre:MDT0001 kilobytes 200000 options -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0001 -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds2_flakey 200000k tune2fs 1.47.3-wc2 (11-Nov-2025) Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre total: 3 open/close in 0.09 seconds: 32.95 ops/second create 3 short symlinks total 8 drwxr-xr-x 4 root root 4096 Mar 16 11:58 . drwxr-xr-x 4 root root 0 Mar 16 10:13 .. drwxr-xr-x 2 root root 4096 Mar 16 11:58 d72.conf-sanity lrwxrwxrwx 1 root root 45 Mar 16 11:58 f72.conf-sanity-1 -> /mnt/lustre/d72.conf-sanity/f72.conf-sanity-1 lrwxrwxrwx 1 root root 45 Mar 16 11:58 f72.conf-sanity-2 -> /mnt/lustre/d72.conf-sanity/f72.conf-sanity-2 lrwxrwxrwx 1 root root 45 Mar 16 11:58 f72.conf-sanity-3 -> /mnt/lustre/d72.conf-sanity/f72.conf-sanity-3 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8 oleg350-server: e2fsck 1.47.3-wc2 (11-Nov-2025) oleg350-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2], used inodes 280/79992 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 82 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 84 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 88 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 88 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 89 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 90 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 91 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 92 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 93 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 94 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 94 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 96 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 97 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 98 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 104 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 106 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 108 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 108 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 109 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 109 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 110 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 110 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 122 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 124 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 125 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 125 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 126 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 131 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 131 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 132 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 132 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 133 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 133 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 136 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 148 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 148 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 149 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 149 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 150 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 151 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 152 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 153 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 153 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 157 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 158 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 159 badness 0 to 2 for 10084 [Thread 0] group 0 finished [Thread 0] e2fsck_pass1_run:2316: increase inode 26693 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26693 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 26723 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 26723 badness 1 to 3 for 10084 [Thread 0] group 1 finished [Thread 0] e2fsck_pass1_run:2316: increase inode 53373 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 53373 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 53374 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 53374 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 53375 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 53375 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 53376 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 53376 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 53377 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2316: increase inode 53378 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 53378 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 53379 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 53379 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 53380 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 53380 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 53381 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 53381 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 53382 badness 0 to 1 for 1005c [Thread 0] e2fsck_pass1_run:2680: increase inode 53382 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2316: increase inode 53383 badness 0 to 1 for 1005c [Thread 0] group 2 finished [Thread 0] Pass 1: Memory used: 280k/0k (155k/126k), time: 0.01/ 0.00/ 0.00 [Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 82.43MB/s [Thread 0] Scanned group range [0, 2], used inodes 280/281 Pass 2: Checking directory structure Pass 2: Memory used: 280k/0k (106k/175k), time: 0.01/ 0.00/ 0.00 Pass 2: I/O read: 1MB, write: 0MB, rate: 95.83MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 280k/0k (106k/175k), time: 0.03/ 0.00/ 0.01 Pass 3: Memory used: 280k/0k (105k/176k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 280k/0k (77k/204k), time: 0.01/ 0.00/ 0.00 Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 5: Checking group summary information Pass 5: Memory used: 388k/0k (77k/312k), time: 0.01/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 162.81MB/s 280 inodes used (0.35%, out of 79992) 2 non-contiguous files (0.7%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 204 24541 blocks used (49.08%, out of 50000) 0 bad blocks 1 large file 149 regular files 118 directories 0 character device files 0 block device files 0 fifos 0 links 3 symbolic links (3 fast symbolic links) 0 sockets ------------ 270 files Memory used: 388k/0k (80k/309k), time: 0.05/ 0.01/ 0.01 I/O read: 1MB, write: 0MB, rate: 20.22MB/s PASS 72 (124s) == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:58:43 (1773676723) checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 failover.node=1.2.3.4@tcp start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre failover_nids: [ "192.168.203.150@tcp", "1.2.3.4@tcp" ] umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 failover.node=1.2.3.4@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server PASS 73a (85s) == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:00:08 (1773676808) checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Comment: just a comment for tail oleg350-server: tunefs.lustre: params are too long: oleg350-server: mgsnode=192.168.203.150@tcp:111.222.173.1@tcp,111.222.142.1@tcp:111.222.173.2@tcp,111.222.142.2@tcp:111.222.173.3@tcp,111.222.142.3@tcp:111.222.173.4@tcp,111.222.142.4@tcp:111.222.173.5@tcp,111.222.142.5@tcp:111.222.173.6@tcp,111.222.142.6@tcp:111.222.173.7@tcp,111.222.142.7@tcp:111.222.173.8@tcp,111.222.142.8@tcp:111.222.173.9@tcp,111.222.142.9@tcp:111.222.173.10@tcp,111.222.142.10@tcp:111.222.173.11@tcp,111.222.142.11@tcp:111.222.173.12@tcp,111.222.142.12@tcp:111.222.173.13@tcp,111.222.142.13@tcp:111.222.173.14@tcp,111.222.142.14@tcp:111.222.173.15@tcp,111.222.142.15@tcp:111.222.173.16@tcp,111.222.142.16@tcp:111.222.173.17@tcp,111.222.142.17@tcp:111.222.173.18@tcp,111.222.142.18@tcp:111.222.173.19@tcp,111.222.142.19@tcp:111.222.173.20@tcp,111.222.142.20@tcp:111.222.173.21@tcp,111.222.142.21@tcp:111.222.173.22@tcp,111.222.142.22@tcp:111.222.173.23@tcp,111.222.142.23@tcp:111.222.173.24@tcp,111.222.142.24@tcp:111.222.173.25@tcp,111.222.142.25@tcp:111.222.173.26@tcp,111.222.142.26@tcp:111.222.173.27@tcp,111.222.142.27@tcp:111.222.173.28@tcp,111.222.142.28@tcp:111.222.173.29@tcp,111.222.142.29@tcp:111.222.173.30@tcp,111.222.142.30@tcp:111.222.173.31@tcp,111.222.142.31@tcp:111.222.173.32@tcp,111.222.142.32@tcp:111.222.173.33@tcp,111.222.142.33@tcp:111.222.173.34@tcp,111.222.142.34@tcp:111.222.173.35@tcp,111.222.142.35@tcp:111.222.173.36@tcp,111.222.142.36@tcp:111.222.173.37@tcp,111.222.142.37@tcp:111.222.173.38@tcp,111.222.142.38@tcp:111.222.173.39@tcp,111.222.142.39@tcp:111.222.173.40@tcp,111.222.142.40@tcp:111.222.173.41@tcp,111.222.142.41@tcp:111.222.173.42@tcp,111.222.142.42@tcp:111.222.173.43@tcp,111.222.142.43@tcp:111.222.173.44@tcp,111.222.142.44@tcp:111.222.173.45@tcp,111.222.142.45@tcp:111.222.173.46@tcp,111.222.142.46@tcp:111.222.173.47@tcp,111.222.142.47@tcp:111.222.173.48@tcp,111.222.142.48@tcp:111.222.173.49@tcp,111.222.142.49@tcp:111.222.173.50@tcp,111.222.142.50@tcp sys.timeout=20 failover.node=111.222.173.1@tcp,111.222.142.1@tcp:111.222.173.2@tcp,111.222.142.2@tcp:111.222.173.3@tcp,111.222.142.3@tcp:111.222.173.4@tcp,111.222.142.4@tcp:111.222.173.5@tcp,111.222.142.5@tcp:111.222.173.6@tcp,111.222.142.6@tcp:111.222.173.7@tcp,111.222.142.7@tcp:111.222.173.8@tcp,111.222.142.8@tcp:111.222.173.9@tcp,111.222.142.9@tcp:111.222.173.10@tcp,111.222.142.10@tcp:111.222.173.11@tcp,111.222.142.11@tcp:111.222.173.12@tcp,111.222.142.12@tcp:111.222.173.13@tcp,111.222.142.13@tcp:111.222.173.14@tcp,111.222.142.14@tcp:111.222.173.15@tcp,111.222.142.15@tcp:111.222.173.16@tcp,111.222.142.16@tcp:111.222.173.17@tcp,111.222.142.17@tcp:111.222.173.18@tcp,111.222.142.18@tcp:111.222.173.19@tcp,111.222.142.19@tcp:111.222.173.20@tcp,111.222.142.20@tcp:111.222.173.21@tcp,111.222.142.21@tcp:111.222.173.22@tcp,111.222.142.22@tcp:111.222.173.23@tcp,111.222.142.23@tcp:111.222.173.24@tcp,111.222.142.24@tcp:111.222.173.25@tcp,111.222.142.25@tcp:111.222.173.26@tcp,111.222.142.26@tcp:111.222.173.27@tcp,111.222.142.27@tcp:111.222.173.28@tcp,111.222.142.28@tcp:111.222.173.29@tcp,111.222.142.29@tcp:111.222.173.30@tcp,111.222.142.30@tcp:111.222.173.31@tcp,111.222.142.31@tcp:111.222.173.32@tcp,111.222.142.32@tcp:111.222.173.33@tcp,111.222.142.33@tcp:111.222.173.34@tcp,111.222.142.34@tcp:111.222.173.35@tcp,111.222.142.35@tcp:111.222.173.36@tcp,111.222.142.36@tcp:111.222.173.37@tcp,111.222.142.37@tcp:111.222.173.38@tcp,111.222.142.38@tcp:111.222.173.39@tcp,111.222.142.39@tcp:111.222.173.40@tcp,111.222.142.40@tcp:111.222.173.41@tcp,111.222.142.41@tcp:111.222.173.42@tcp,111.222.142.42@tcp:111.222.173.43@tcp,111.222.142.43@tcp:111.222.173.44@tcp,111.222.142.44@tcp:111.222.173.45@tcp,111.222.142.45@tcp:111.222.173.46@tcp,111.222.142.46@tcp:111.222.173.47@tcp,111.222.142.47@tcp:111.222.173.48@tcp,111.222.142.48@tcp:111.222.173.49@tcp,111.222.142.49@tcp:111.222.173.50@tcp,111.222.142.50@tcp +failover.node==111.222.173.51@tcp,111.222.142.51@tcp oleg350-server: tunefs.lustre: exiting with 7 (Argument list too long) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 7 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server Found total 201 NIDs in parameters Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 73b (662s) == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:11:10 (1773677470) umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server stop mds service on oleg350-server stop mds service on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg350-server: sh: /usr/sbin/ksocklnd-config: No such file or directory restart with 35 nets start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. PASS 73c (151s) == conf-sanity test 73d: erase + new parameter doesn't corrupt mountdata ========================================================== 12:13:42 (1773677622) checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=1 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=1 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=10.23.7.1@tcp:10.23.7.2@tcp Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 73d (26s) == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:14:08 (1773677648) umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server stop mds service on oleg350-server stop mds service on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing lnet_if_list Node: oleg350-server, rinf=ens2 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 oleg350-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.150 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp50 local NI(s): - nid: 192.168.203.50@tcp50 status: up interfaces: 0: ens2 Try to mount to MGS NID 192.168.203.150@tcp50 stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.50 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.50 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp50 local NI(s): - nid: 192.168.203.50@tcp50 status: up interfaces: 0: ens2 Try to mount after MGS remount setup single mount lustre success UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1616 80976 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1492 81100 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1528 126688 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1528 126688 2% /mnt/lustre umount lustre on /mnt/lustre..... stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. PASS 73e (214s) == conf-sanity test 74: Test per-device adaptive timeout parameters ========================================================== 12:17:42 (1773677862) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Permanent disk data: Target: fs15246:MDT0000 Index: 0 Lustre FS: fs15246 Mount type: ldiskfs Flags: 0x61 (MDT first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity formatting backing filesystem ldiskfs on /dev/loop0 target name fs15246:MDT0000 kilobytes 200000 options -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L fs15246:MDT0000 -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/loop0 200000k pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Permanent disk data: Target: fs15246:OST0000 Index: 0 Lustre FS: fs15246 Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 formatting backing filesystem ldiskfs on /dev/loop0 target name fs15246:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L fs15246:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/loop0 200000k /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1837: OST_INDICES: bad array subscript pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Start fs2ost: mount -t lustre -o localrecov /dev/mapper/fs2ost_flakey /mnt/lustre-fs2ost seq.cli-fs15246-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /tmp/lustre-ost1_2 Started fs15246-OST0000 /mnt/lustre-fs2ost: 83.1 MiB (87142400 bytes) trimmed pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Start fs2mds: mount -t lustre -o localrecov /dev/mapper/fs2mds_flakey /mnt/lustre-fs2mds oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /tmp/lustre-mdt1_2 Started fs15246-MDT0000 ========== All mounted lustre fs ==================== 192.168.203.150@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) 192.168.203.150@tcp:/fs15246 on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose) ===================================================== osc.lustre-OST0000-osc-ffff9cc4c6254000.ldlm_enqueue_min=99 obdfilter.lustre-OST0000.at_min=1 mdc.lustre-MDT0000-mdc-ffff9cc4c6254000.at_max=599 mdc.lustre-MDT0001-mdc-ffff9cc4c6254000.at_max=599 mdt.lustre-MDT0000.at_max=599 mdt.lustre-MDT0001.at_max=599 obdfilter.lustre-OST0000.at_max=599 mgc.MGC192.168.203.150@tcp.at_history=588 mgs.MGS.at_history=588 mgs.MGS.at_history=0 mgc.MGC192.168.203.150@tcp.at_history=0 mdt.lustre-MDT0000.at_max=0 mdt.lustre-MDT0001.at_max=0 obdfilter.lustre-OST0000.at_max=0 mdc.lustre-MDT0000-mdc-ffff9cc4c6254000.at_max=0 mdc.lustre-MDT0001-mdc-ffff9cc4c6254000.at_max=0 obdfilter.lustre-OST0000.at_min=0 osc.lustre-OST0000-osc-ffff9cc4c6254000.ldlm_enqueue_min=0 umount /mnt/lustre2 ... stopping fs2mds ... Stopping /mnt/lustre-fs2mds (opts:-f) on oleg350-server stopping fs2ost ... Stopping /mnt/lustre-fs2ost (opts:-f) on oleg350-server umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. PASS 74 (107s) == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 12:19:29 (1773677969) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre-MDT0000 kilobytes 200000 options -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDT0000 -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre-MDT0000 kilobytes 200000 options -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDT0000 -b 4096 -I 1024 -i 2560 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server PASS 75 (74s) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 9821 sec ========== 12:21:01 (1773678061) === conf-sanity: start cleanup 12:21:01 (1773678061) === === conf-sanity: finish cleanup 12:21:02 (1773678062) ===