-----============= acceptance-small: conf-sanity ============----- Tue Sep 8 17:34:50 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg339-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex - see CLIENT_VERSION > v2_15_63-53-g1a6ef725c2 (34682650 > 34553653) for LU-16938, go 81 excepting tests: 55 63 67 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg339-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:37:29 (1788903449) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Permanent disk data: Device: lustre-mdt1_2/mdt1_2 Target: test-33a:MDT0000 Index: 0 Lustre FS: test-33a Mount type: zfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1_2/mdt1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=101 lustre:index=0 lustre:fsname=test-33a lustre:svname=test-33a:MDT0000 Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: test-33a:OST1fff Index: 8191 Lustre FS: test-33a Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp:192.168.203.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.139@tcp:192.168.203.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=8191 lustre:fsname=test-33a lustre:svname=test-33a:OST1fff Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1_2/mdt1_2 Started test-33a-MDT0000 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1849: OST_INDICES: bad array subscript Start fs2ost: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-test-33a-OST1fff-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started test-33a-OST1fff ok. /mnt/lustre2/hosts lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 8191 obdidx objid objid group 8191 2 0x2 0x240000400 config_log: test-33a-MDT0000 cancel catalog test-33a-MDT0000 log_idx 20: done cancel catalog test-33a-MDT0000 log_idx 19: done cancel catalog test-33a-MDT0000 log_idx 18: done del_ost: cancelled 3 catalog entries config_log: test-33a-client cancel catalog test-33a-client log_idx 25: done cancel catalog test-33a-client log_idx 24: done cancel catalog test-33a-client log_idx 23: done del_ost: cancelled 3 catalog entries ok. Stopping /mnt/lustre-fs2ost (opts:-f) on oleg339-server Stopping /mnt/lustre-fs2mds (opts:-f) on oleg339-server stop ost1 service on oleg339-server stop mds service on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 umount /mnt/lustre2 ... umount: /mnt/lustre2: not mounted. stopping fs2mds ... stopping fs2ost ... PASS 33a (68s) == conf-sanity test 33b: Drop cancel during umount ======= 17:38:38 (1788903518) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865991 s, 12.1 MB/s fail_loc=0x80000304 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 PASS 33b (79s) == conf-sanity test 33c: Mount ost with a large index number ========================================================== 17:39:56 (1788903596) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Permanent disk data: Device: lustre-mdt1_2/mdt1_2 Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1_2/mdt1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=101 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:MDT0000 Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: lustre:OST07c6 Index: 1990 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp:192.168.203.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.139@tcp:192.168.203.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=1990 lustre:fsname=lustre lustre:svname=lustre:OST07c6 Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1_2/mdt1_2 Started lustre-MDT0000 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1849: OST_INDICES: bad array subscript Start fs2ost: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-lustre-OST07c6-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST07c6 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Creating new pool qpool1 oleg339-server: Pool lustre.qpool1 created Adding targets to pool oleg339-server: OST lustre-OST07c6_UUID added to pool lustre.qpool1 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.85226 s, 10.8 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.109081 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.52977 s, 13.0 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.112068 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.65394 s, 12.0 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.106414 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.0798788 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 20+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.13812 s, 17.5 MB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.099099 s, 0.0 kB/s running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct] dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded 1+0 records in 0+0 records out 0 bytes copied, 0.0900579 s, 0.0 kB/s Destroy the created pools: qpool1 lustre.qpool1 oleg339-server: OST lustre-OST07c6_UUID removed from pool lustre.qpool1 oleg339-server: Pool lustre.qpool1 destroyed Waiting 90s for 'foo' Updated after 5s: want 'foo' got 'foo' umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-fs2ost (opts:-f) on oleg339-server Stopping /mnt/lustre-fs2mds (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. PASS 33c (150s) == conf-sanity test 33d: Don't panic when enable project quota ========================================================== 17:42:26 (1788903746) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre lfs setquota: warning: inode hardlimit '1024' smaller than minimum qunit size See 'lfs help setquota' or Lustre manual for details osd-zfs.lustre-MDT0000.quota_slave.enabled=p osd-zfs.lustre-OST0000.quota_slave.enabled=p umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. PASS 33d (80s) == conf-sanity test 34a: umount with opened file should be fail ========================================================== 17:43:46 (1788903826) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre multiop /mnt/lustre/file vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.24935 manual umount lustre on /mnt/lustre.... umount: /mnt/lustre: target is busy. umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 34a (60s) == conf-sanity test 34b: force umount with failed mds should be normal ========================================================== 17:44:46 (1788903886) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server manual umount lustre on /mnt/lustre.... umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 34b (70s) == conf-sanity test 34c: force umount with failed ost should be normal ========================================================== 17:45:56 (1788903956) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server manual umount lustre on /mnt/lustre.... umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 34c (67s) == conf-sanity test 35a: Reconnect to the last active server first ========================================================== 17:47:03 (1788904023) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre debug=ha Set up a fake failnode for the MDS Wait for RECONNECT_INTERVAL seconds (10s) conf-sanity.sh test_35a 2026-09-0817h47m53s Stopping the MDT: lustre-MDT0000 stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Restarting the MDT: lustre-MDT0000 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 Wait for df (34687) ... done debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout Debug log: 75 lines, 75 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 35a (199s) == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 17:50:22 (1788904222) start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2026-09-0817h51m23s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff9a49040a9000.stats=clear Creating a test file and stat it File: /mnt/lustre/d35b.conf-sanity/f35b.conf-sanity Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 17:52:18.000000000 -0400 Modify: 2026-09-08 17:52:18.000000000 -0400 Change: 2026-09-08 17:52:18.000000000 -0400 Birth: 2026-09-08 17:52:18.000000000 -0400 Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 44 lines, 44 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 35b (301s) == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 17:55:23 (1788904523) SKIP: conf-sanity test_36 remote OST SKIP 36 (3s) == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 17:55:26 (1788904526) SKIP: conf-sanity test_37 ldiskfs only test SKIP 37 (3s) == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:55:30 (1788904530) start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server 000000 2 000008 oleg339-server: 8+0 records in oleg339-server: 8+0 records out oleg339-server: 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00112606 s, 3.6 MB/s start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 38 (208s) == conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ========================================================== 17:58:57 (1788904737) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug +malloc all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug +malloc all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 39 (91s) == conf-sanity test 40: race during service thread startup ========================================================== 18:00:28 (1788904828) start ost1 service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 fail_loc=0x80000706 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 PASS 40 (118s) == conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================================== 18:02:26 (1788904946) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov -o nosvc -n lustre-mdt1/mdt1 /mnt/lustre-mds1 nomtab: 1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov -o nomgs,force lustre-mdt1/mdt1 /mnt/lustre-mds1 force: 1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre blah blah umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 41a (67s) == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:03:33 (1788905013) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov -o nosvc -n lustre-mdt1/mdt1 /mnt/lustre-mds1 nomtab: 1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Commit the device label on lustre-mdt1/mdt1 Started lustre:MDT0000 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov -o nomgs,force lustre-mdt1/mdt1 /mnt/lustre-mds1 force: 1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre blah blah umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 41b (106s) == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:05:19 (1788905119) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' fail_loc=0x80000716 oleg339-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: Operation already in progress oleg339-server: The target service is already running. (lustre-mdt1/mdt1) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 114 fail_loc=0x0 1st MDT start failed with 114 2nd MDT start succeed oleg339-server: oleg339-server.virtnet: executing lsmod fail_loc=0x80000716 oleg339-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Operation already in progress oleg339-server: The target service is already running. (lustre-ost1/ost1) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 114 fail_loc=0x0 1st OST start succeed 2nd OST start failed with 114 stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 PASS 41c (129s) == conf-sanity test 42: allow client/server mount/unmount with invalid config param ========================================================== 18:07:28 (1788905248) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 PASS 42 (77s) == conf-sanity test 43a: check root_squash and nosquash_nids ========================================================== 18:08:45 (1788905325) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Setting lustre.mdt.root_squash from 0:0 to 0:0 Setting lustre.mdt.nosquash_nids from NONE to NONE Setting lustre.mdt.root_squash from 0:0 to 500:500 Waiting 90s for '500:500' Updated after 7s: want '500:500' got '500:500' /mnt/lustre/f43a.conf-sanity-userfile: owner uid 500 (-rw-------): root read permission is granted - ok /mnt/lustre/f43a.conf-sanity-userfile: owner uid 500 (-rw-------): root write permission is granted - ok /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root read permission is denied - ok /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root write permission is denied - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root unlink permission is denied - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root create permission is denied - ok /mnt/lustre/f43a.conf-sanity-user1file: owner uid 501 (-rw-------): root read permission is denied - ok /mnt/lustre/f43a.conf-sanity-user1file: owner uid 501 (-rw-------): root write permission is denied - ok /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4333: 76540 Terminated runas -u $ID1 tail -f $DIR/$tfile-user1file > /dev/null 2>&1 Setting lustre-MDTall.mdt.nosquash_nids from NONE to 2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp Waiting 90s for '2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp' Updated after 4s: want '2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp' got '2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp' Waiting 90s for '2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp' Updated after 2s: want '2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp' got '2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp' /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root read permission is granted - ok /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root write permission is granted - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root unlink permission is granted - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root create permission is granted - ok Setting lustre-MDTall.mdt.nosquash_nids from 2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp to 192.168.203.39/24@tcp Waiting 90s for '192.168.203.39/24@tcp' Updated after 6s: want '192.168.203.39/24@tcp' got '192.168.203.39/24@tcp' Waiting 90s for '192.168.203.39/24@tcp' Updated after 2s: want '192.168.203.39/24@tcp' got '192.168.203.39/24@tcp' /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root read permission is granted - ok /mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root write permission is granted - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root unlink permission is granted - ok /mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root create permission is granted - ok umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 43a (70s) == conf-sanity test 43b: parse nosquash_nids with commas in expr_list ========================================================== 18:09:55 (1788905395) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' 192.168.203.39: Warning: Permanently added '192.168.203.39' (ECDSA) to the list of known hosts. Permanent disk data: Device: lustre-mdt1_2/mdt1_2 Target: MGS Index: unassigned Lustre FS: test1234 Mount type: zfs Flags: 0x64 (MGS first_time update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.root_squash=500:500 mdt.nosquash_nids=192.168.203.[39,39,39]@tcp mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1_2/mdt1_2 properties lustre:sys.timeout=20 lustre:mdt.root_squash=500:500 lustre:mdt.nosquash_nids=192.168.203.[39,39,39]@tcp lustre:version=1 lustre:flags=100 lustre:index=65535 lustre:fsname=test1234 lustre:svname=MGS Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started MGS Stopping /mnt/lustre-fs2mds (opts:-f) on oleg339-server umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 PASS 43b (29s) == conf-sanity test 44: mounted client proc entry exists ========================================================== 18:10:24 (1788905424) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 44 (46s) SKIP: conf-sanity test_45 skipping SLOW test 45 == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 18:11:11 (1788905471) Testing with 2 OSTs Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a49182f1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a49182f1000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 0 34 0x22 0x240000400 File: /mnt/lustre/widestripe Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 18:12:26.000000000 -0400 Modify: 2026-09-08 18:12:26.000000000 -0400 Change: 2026-09-08 18:12:26.000000000 -0400 Birth: 2026-09-08 18:12:26.000000000 -0400 umount lustre on /mnt/lustre2..... Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop ost1 service on oleg339-server stop mds service on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 46a (198s) == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 18:14:29 (1788905669) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success ldlm.namespaces.lustre-MDT0000-mdc-ffff9a4904a73800.lru_size=100 ldlm.namespaces.lustre-OST0000-osc-ffff9a4904a73800.lru_size=100 ldlm.namespaces.lustre-MDT0000-mdc-ffff9a4904a73800.lru_size=100 Failing ost1 on oleg339-server Stopping /mnt/lustre-ost1 (opts:) on oleg339-server 18:15:22 (1788905722) shut down facet: ost1 facet_host: oleg339-server facet_failover_host: oleg339-server Failover ost1 to oleg339-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 18:15:39 (1788905739) targets are mounted 18:15:39 (1788905739) facet_failover done Failing mds1 on oleg339-server Stopping /mnt/lustre-mds1 (opts:) on oleg339-server 18:15:41 (1788905741) shut down facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server Failover mds1 to oleg339-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 18:15:58 (1788905758) targets are mounted 18:15:58 (1788905758) facet_failover done ldlm.namespaces.lustre-MDT0000-mdc-ffff9a4904a73800.lru_size=100 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 47a (111s) == conf-sanity test 47b: client restart does not lose lru_resize ========================================================== 18:16:20 (1788905780) Failing mds1 on oleg339-server 18:16:22 (1788905782) shut down facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server Failover mds1 to oleg339-server mount facets: mds1 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 18:16:54 (1788905814) targets are mounted 18:16:54 (1788905814) facet_failover done pdsh@oleg339-client: oleg339-client: ssh exited with exit code 95 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 95 Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success ln: failed to create symbolic link '/home/green/git/lustre-release/lustre/utils/lctl': File exists SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl SKIP 47b (111s) == conf-sanity test 48: too many acls on file ============ 18:18:11 (1788905891) start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg339-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of lustre-ost1/ost1 on ost1 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab 192.168.203.139@tcp:/lustre is already mounted on /mnt/lustre setup single mount lustre success debug=0 debug=0 /mnt/lustre stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/widestripe lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4 0x4 0x240000400 It is expected to hold at least 4500 ACL entries File: /mnt/lustre/widestripe Size: 3 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272502275 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 18:18:24.000000000 -0400 Modify: 2026-09-08 18:18:24.000000000 -0400 Change: 2026-09-08 18:21:16.000000000 -0400 Birth: 2026-09-08 18:18:24.000000000 -0400 getfacl: Removing leading '/' from absolute path names Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg339-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 48 (289s) == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:23:00 (1788906180) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success check ldlm_timout... umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 49a (71s) == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:24:11 (1788906251) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 49b (82s) == conf-sanity test 50a: lazystatfs all servers available ========================================================== 18:25:33 (1788906333) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre llite.lustre-ffff9a490640e000.lazystatfs=1 multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7612 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 2048 379520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 2048 378880 1% /mnt/lustre[OST:0] filesystem_summary: 382976 2048 378880 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4731: kill: (119970) - No such process umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 50a (82s) == conf-sanity test 50b: lazystatfs all servers down ===== 18:26:55 (1788906415) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre llite.lustre-ffff9a49124c9800.lazystatfs=1 stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server oleg339-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9a49124c9800.ost_server_uuid 50 0 UP mgc MGC192.168.203.139@tcp 98cd4047-a4da-40c0-bfca-fcaf20ef8adf 5 1 UP lov lustre-clilov-ffff9a49124c9800 09454666-073f-4dd1-9dd9-c1318caf190c 4 2 UP lmv lustre-clilmv-ffff9a49124c9800 09454666-073f-4dd1-9dd9-c1318caf190c 5 3 UP mdc lustre-MDT0000-mdc-ffff9a49124c9800 09454666-073f-4dd1-9dd9-c1318caf190c 5 4 UP osc lustre-OST0000-osc-ffff9a49124c9800 09454666-073f-4dd1-9dd9-c1318caf190c 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7612 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 2048 379520 1% /mnt/lustre[MDT:0] filesystem_summary: 383616 2048 379520 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4731: kill: (123459) - No such process umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 50b (137s) == conf-sanity test 50c: lazystatfs one server down ====== 18:29:12 (1788906552) start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 start ost2 service on oleg339-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre llite.lustre-ffff9a4902ece800.lazystatfs=1 stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server oleg339-server: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7612 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 2048 379520 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 2048 378880 1% /mnt/lustre[OST:1] filesystem_summary: 382976 2048 378880 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4731: kill: (125642) - No such process umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 50c (223s) == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:32:55 (1788906775) start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 start ost2 service on oleg339-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre llite.lustre-ffff9a4905fd2800.lazystatfs=1 stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7612 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 2048 379520 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 2048 378880 1% /mnt/lustre[OST:1] filesystem_summary: 382976 2048 378880 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4731: kill: (130812) - No such process umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 50d (150s) == conf-sanity test 50e: normal statfs all servers down == 18:35:25 (1788906925) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server oleg339-server: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre llite.lustre-ffff9a4903491000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7612 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 50e (155s) == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:38:00 (1788907080) start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg339-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server oleg339-server: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre llite.lustre-ffff9a4905b7c000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7612 start ost2 service on oleg339-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0001 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 50f (170s) == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:40:50 (1788907250) start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre start ost2 service on oleg339-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a49083f1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a49083f1800.ost_server_uuid in IDLE state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.139@tcp:/lustre 382976 2048 378880 1% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 50g (202s) == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:44:12 (1788907452) checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.139@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.139@tcp osc.active=0 autodegrade=on Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.203.139@tcp lustre:osc.active=0 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 start ost2 service on oleg339-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Setting lustre-OST0000.osc.active from 0 to 1 Waiting 90s for '1' Updated after 7s: want '1' got '1' create a file after OST1 is activated 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.8998 s, 11.7 MB/s umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 50h (110s) == conf-sanity test 50i: activate deactivated MDT ======== 18:46:02 (1788907562) SKIP: conf-sanity test_50i needs >= 2 MDTs SKIP 50i (3s) == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:46:05 (1788907565) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg339-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 51 (196s) == conf-sanity test 52: check recovering objects from lost+found ========================================================== 18:49:21 (1788907761) SKIP: conf-sanity test_52 ldiskfs only test SKIP 52 (1s) == conf-sanity test 53a: check OSS thread count params === 18:49:22 (1788907762) start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success checking (($tstarted && $tmin && $tmax)) (((3 && 3 && 116)))... checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((3 >= 3 && 3 <= 116 )))... ost.OSS.ost.threads_min=4 ost.OSS.ost.threads_max=115 checking (($tmin2 == ($tmin + $nthrs) && $tmax2 == ($tmax - $nthrs))) (((4 == (3 + 1) && 115 == (116 - 1))))... ost.OSS.ost.threads_max=4 checking (($tmin2 == $tmin && $tmax2 == $tmin)) (((4 == 4 && 4 == 4)))... oleg339-server: error: set_param: setting /sys/fs/lustre/ost/OSS/ost/threads_max=3: Numerical result out of range oleg339-server: error: set_param: setting 'ost/OSS/ost/threads_max'='3': Numerical result out of range pdsh@oleg339-client: oleg339-server: ssh exited with exit code 34 checking (($tmin2 <= $tmax2)) (((4 <= 4)))... umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. setmodopts: MODOPTS_OST= oss_num_threads=16 start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ost/ost options: 'oss_num_threads=16' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg339-server: ost/ost options: 'oss_num_threads=16' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((4 >= 3 && 4 <= 116 )))... umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 53a (91s) == conf-sanity test 53b: check MDS thread count params === 18:50:53 (1788907853) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success checking (($tstarted && $tmin && $tmax)) (((4 && 3 && 116)))... checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((4 >= 3 && 4 <= 116 )))... mds.MDS.mdt..threads_min=4 mds.MDS.mdt..threads_max=115 checking (($tmin2 == ($tmin + $nthrs) && $tmax2 == ($tmax - $nthrs))) (((4 == (3 + 1) && 115 == (116 - 1))))... mds.MDS.mdt..threads_max=4 checking (($tmin2 == $tmin && $tmax2 == $tmin)) (((4 == 4 && 4 == 4)))... oleg339-server: error: set_param: setting /sys/fs/lustre/mds/MDS/mdt//threads_max=3: Numerical result out of range oleg339-server: error: set_param: setting 'mds/MDS/mdt//threads_max'='3': Numerical result out of range pdsh@oleg339-client: oleg339-server: ssh exited with exit code 34 checking (($tmin2 <= $tmax2)) (((4 <= 4)))... umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. setmodopts: MODOPTS_MDT=mdt_enable_flr_ec=1 mds_num_threads=16 start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1 mds_num_threads=16' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1 mds_num_threads=16' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success setmodopts: MODOPTS_MDT=mdt_enable_flr_ec=1 checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((16 >= 16 && 16 <= 16 )))... umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 PASS 53b (94s) == conf-sanity test 54a: test llverdev and partial verify of device ========================================================== 18:52:27 (1788907947) SKIP: conf-sanity test_54a ldiskfs only test SKIP 54a (1s) == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:52:28 (1788907948) SKIP: conf-sanity test_54b ldiskfs only test SKIP 54b (1s) SKIP: conf-sanity test_55 skipping excluded test 55 == conf-sanity test 56a: check big OST indexes and out-of-index-order start ========================================================== 18:52:30 (1788907950) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST2710 Index: 10000 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=10000 lustre:fsname=lustre lustre:svname=lustre:OST2710 Permanent disk data: Device: lustre-ost2/ost2 Target: lustre:OST03e8 Index: 1000 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost2 /dev/vdf mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2/ost2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2/ost2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=1000 lustre:fsname=lustre lustre:svname=lustre:OST03e8 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST2710-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST2710 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 start ost2 service on oleg339-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST03e8-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST03e8 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre ok OBDS: 1000: lustre-OST03e8_UUID ACTIVE 10000: lustre-OST2710_UUID ACTIVE 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00459702 s, 891 kB/s oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:) Stopping client oleg339-client.virtnet /mnt/lustre opts: Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 56a (119s) == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:54:29 (1788908069) SKIP: conf-sanity test_56b needs >= 3 MDTs SKIP 56b (1s) == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:54:31 (1788908071) oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp autodegrade=on sys.timeout=20 failover.node=192.168.203.139@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.139@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg339-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Cannot assign requested address pdsh@oleg339-client: oleg339-server: ssh exited with exit code 99 oleg339-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg339-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 Start of lustre-ost1/ost1 on ost1 failed 99 umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 57a (69s) == conf-sanity test 57b: initial registration from servicenode should not fail ========================================================== 18:55:40 (1788908140) oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x1042 (OST update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on failover.node=192.168.203.139@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:failover.node=192.168.203.139@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4162 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 PASS 57b (92s) == conf-sanity test 57c: servicenode metadata preserves legacy discovery by default ========================================================== 18:57:12 (1788908232) SERVICENODE DEBUG: begin legacy-policy test umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 SERVICENODE DEBUG: setup base_policy=none servicenode_policy=disabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg339-server: oleg339-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory SERVICENODE DEBUG: net=tcp nid='192.168.203.139@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.139@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: temporary fsname='sv22463' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv22463:OST0000 Index: 0 Lustre FS: sv22463 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv22463 lustre:svname=sv22463:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv22463:OST0001 Index: 1 Lustre FS: sv22463 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp1 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost2_2 /tmp/lustre-ost2_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv22463 lustre:svname=sv22463:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv22463-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv22463-OST0000 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv22463-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv22463-OST0001 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: legacy expected full view='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: ping seed=192.168.203.139@tcp present='192.168.203.139@tcp,192.168.203.139@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: discover seed=192.168.203.139@tcp present='192.168.203.139@tcp,192.168.203.139@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: legacy-policy test complete PASS 57c (106s) == conf-sanity test 57d: servicenode policy splits OSS discovery views ========================================================== 18:58:58 (1788908338) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg339-server: oleg339-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 SERVICENODE DEBUG: net=tcp nid='192.168.203.139@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.139@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv18553' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv18553:OST0000 Index: 0 Lustre FS: sv18553 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv18553 lustre:svname=sv18553:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv18553:OST0001 Index: 1 Lustre FS: sv18553 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv18553 lustre:svname=sv18553:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv18553-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv18553-OST0000 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv18553-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv18553-OST0001 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.203.139@tcp present='192.168.203.139@tcp' absent='192.168.203.139@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp SERVICENODE DEBUG: ping seed=192.168.203.139@tcp1 present='192.168.203.139@tcp1' absent='192.168.203.139@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: discover seed=192.168.203.139@tcp present='192.168.203.139@tcp' absent='192.168.203.139@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: discover seed=192.168.203.139@tcp1 present='192.168.203.139@tcp1' absent='192.168.203.139@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp1 Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp1 state: NA PASS 57d (104s) == conf-sanity test 57e: servicenode policy overrides self grouping ========================================================== 19:00:42 (1788908442) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode setmodopts: MODOPTS_LNET= lnet_self_nets=tcp,tcp1 SERVICENODE DEBUG: setup base_policy=self servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET=' lnet_self_nets=tcp,tcp1' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'lnet_self_nets=tcp,tcp1' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ../lnet/lnet/lnet options: 'lnet_self_nets=tcp,tcp1' oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg339-server: oleg339-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 SERVICENODE DEBUG: net=tcp nid='192.168.203.139@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.139@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_self_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv8022' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv8022:OST0000 Index: 0 Lustre FS: sv8022 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp,192.168.203.139@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp,192.168.203.139@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv8022 lustre:svname=sv8022:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv8022:OST0001 Index: 1 Lustre FS: sv8022 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv8022 lustre:svname=sv8022:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv8022-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv8022-OST0000 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_self_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: ping seed=192.168.203.139@tcp present='192.168.203.139@tcp,192.168.203.139@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: ping seed=192.168.203.139@tcp1 present='192.168.203.139@tcp,192.168.203.139@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: discover seed=192.168.203.139@tcp present='192.168.203.139@tcp,192.168.203.139@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: discover seed=192.168.203.139@tcp1 present='192.168.203.139@tcp,192.168.203.139@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA PASS 57e (95s) == conf-sanity test 57f: servicenode policy overrides NUMA grouping ========================================================== 19:02:17 (1788908537) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode setmodopts: MODOPTS_LNET= lnet_group_numa_nets=tcp,tcp1 SERVICENODE DEBUG: setup base_policy=numa servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET=' lnet_group_numa_nets=tcp,tcp1' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'lnet_group_numa_nets=tcp,tcp1' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ../lnet/lnet/lnet options: 'lnet_group_numa_nets=tcp,tcp1' oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded fail_loc=0xe005 fail_val=2 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg339-server: oleg339-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 SERVICENODE DEBUG: net=tcp nid='192.168.203.139@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.139@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_group_numa_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv7635' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv7635:OST0000 Index: 0 Lustre FS: sv7635 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp,192.168.203.139@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp,192.168.203.139@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv7635 lustre:svname=sv7635:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv7635:OST0001 Index: 1 Lustre FS: sv7635 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv7635 lustre:svname=sv7635:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv7635-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv7635-OST0000 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_group_numa_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: ping seed=192.168.203.139@tcp present='192.168.203.139@tcp,192.168.203.139@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: ping seed=192.168.203.139@tcp1 present='192.168.203.139@tcp,192.168.203.139@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: discover seed=192.168.203.139@tcp present='192.168.203.139@tcp,192.168.203.139@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: discover seed=192.168.203.139@tcp1 present='192.168.203.139@tcp,192.168.203.139@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA PASS 57f (164s) == conf-sanity test 57g: servicenode policy keeps ungrouped NIDs discoverable ========================================================== 19:05:01 (1788908701) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg339-server: oleg339-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 SERVICENODE DEBUG: net=tcp nid='192.168.203.139@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.139@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv14319' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv14319:OST0000 Index: 0 Lustre FS: sv14319 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv14319 lustre:svname=sv14319:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv14319:OST0001 Index: 1 Lustre FS: sv14319 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv14319 lustre:svname=sv14319:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv14319-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv14319-OST0000 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.203.139@tcp1 present='192.168.203.139@tcp1' absent='192.168.203.139@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: discover seed=192.168.203.139@tcp1 present='192.168.203.139@tcp1' absent='192.168.203.139@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp1 Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: ping seed=192.168.203.139@tcp present='192.168.203.139@tcp' absent='192.168.203.139@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp SERVICENODE DEBUG: starting temporary OST2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv14319-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv14319-OST0001 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SERVICENODE DEBUG: after second target startup SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: false peer ni: - nid: 192.168.203.139@tcp state: NA - primary nid: 192.168.203.139@tcp1 Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA peer: - primary nid: 192.168.203.39@tcp1 Multi-Rail: true peer ni: - nid: 192.168.203.39@tcp1 state: NA peer: - primary nid: 192.168.203.39@tcp Multi-Rail: false peer ni: - nid: 192.168.203.39@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.203.139@tcp present='192.168.203.139@tcp' absent='192.168.203.139@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp SERVICENODE DEBUG: ping seed=192.168.203.139@tcp1 present='192.168.203.139@tcp1' absent='192.168.203.139@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: discover seed=192.168.203.139@tcp present='192.168.203.139@tcp' absent='192.168.203.139@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: discover seed=192.168.203.139@tcp1 present='192.168.203.139@tcp1' absent='192.168.203.139@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp1 Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp1 state: NA PASS 57g (169s) == conf-sanity test 57h: servicenode policy is active before the first target mounts ========================================================== 19:07:50 (1788908870) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg339-server: oleg339-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 SERVICENODE DEBUG: net=tcp nid='192.168.203.139@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.139@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: stopping setup after LNet config SERVICENODE DEBUG: ping seed=192.168.203.139@tcp present='192.168.203.139@tcp' absent='192.168.203.139@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp SERVICENODE DEBUG: ping seed=192.168.203.139@tcp1 present='192.168.203.139@tcp1' absent='192.168.203.139@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: discover seed=192.168.203.139@tcp present='192.168.203.139@tcp' absent='192.168.203.139@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: discover seed=192.168.203.139@tcp1 present='192.168.203.139@tcp1' absent='192.168.203.139@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp1 Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp1 state: NA PASS 57h (116s) == conf-sanity test 57i: servicenode Push never falls back to the merged view ========================================================== 19:09:46 (1788908986) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg339-server: oleg339-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 SERVICENODE DEBUG: net=tcp nid='192.168.203.139@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.139@tcp1' SERVICENODE DEBUG: net=tcp2 nid='192.168.203.139@tcp2' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1,tcp2' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1,192.168.203.139@tcp2' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.203.39@tcp2 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.203.139@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv12908' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv12908:OST0000 Index: 0 Lustre FS: sv12908 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp,192.168.203.139@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp,192.168.203.139@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv12908 lustre:svname=sv12908:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv12908:OST0001 Index: 1 Lustre FS: sv12908 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv12908 lustre:svname=sv12908:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv12908-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv12908-OST0000 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1,tcp2' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1,192.168.203.139@tcp2' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.203.39@tcp2 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.203.139@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: discover seed=192.168.203.139@tcp present='192.168.203.139@tcp' absent='192.168.203.139@tcp2' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: starting temporary OST2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv12908-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv12908-OST0001 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SERVICENODE DEBUG: after second target startup SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1,tcp2' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1,192.168.203.139@tcp2' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.203.39@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.203.139@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA - nid: 192.168.203.139@tcp1 state: NA peer: - primary nid: 192.168.203.39@tcp Multi-Rail: true peer ni: - nid: 192.168.203.39@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.203.139@tcp present='192.168.203.139@tcp' absent='192.168.203.139@tcp2' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp - nid: 192.168.203.139@tcp1 PASS 57i (147s) == conf-sanity test 57j: failnode targets contribute no servicenode group ========================================================== 19:12:13 (1788909133) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg339-server: oleg339-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 SERVICENODE DEBUG: net=tcp nid='192.168.203.139@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.139@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv17292' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv17292:OST0000 Index: 0 Lustre FS: sv17292 Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.39@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.39@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=sv17292 lustre:svname=sv17292:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv17292:OST0001 Index: 1 Lustre FS: sv17292 Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.39@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.39@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=1 lustre:fsname=sv17292 lustre:svname=sv17292:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv17292-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv17292-OST0000 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv17292-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv17292-OST0001 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp1 state: NA - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.203.139@tcp present='192.168.203.139@tcp' absent='192.168.203.139@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp SERVICENODE DEBUG: ping seed=192.168.203.139@tcp1 present='192.168.203.139@tcp1' absent='192.168.203.139@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.203.139@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.203.139@tcp1 SERVICENODE DEBUG: discover seed=192.168.203.139@tcp present='192.168.203.139@tcp' absent='192.168.203.139@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.139@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA PASS 57j (128s) == conf-sanity test 57k: unknown target_nid_policy stops the module load ========================================================== 19:14:21 (1788909261) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. oleg339-server: modprobe: FATAL: Module ptlrpc not found in directory /lib/modules/4.18.0rh8.10-debug pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. PASS 57k (14s) == conf-sanity test 57ka: IR READY respects servicenode target NID policy ========================================================== 19:14:35 (1788909275) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg339-server: oleg339-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 SERVICENODE DEBUG: net=tcp nid='192.168.203.139@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.139@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv13596' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv13596:OST0000 Index: 0 Lustre FS: sv13596 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv13596 lustre:svname=sv13596:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv13596:OST0001 Index: 1 Lustre FS: sv13596 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 failover.node=192.168.203.139@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.139@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv13596 lustre:svname=sv13596:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv13596-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv13596-OST0000 oleg339-server: missing pool name argument oleg339-server: usage: oleg339-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg339-client.virtnet OSS=oleg339-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.139@tcp,192.168.203.139@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.39@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.39@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.139@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.139@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.139@tcp Multi-Rail: true peer ni: - nid: 192.168.203.139@tcp state: NA Stopping /mnt/lustre-fs2ost1 (opts:-f) on oleg339-server fail_val=1 fail_loc=0x80000727 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv13596-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started sv13596-OST0000 fail_loc=0 fail_val=0 PASS 57ka (138s) == conf-sanity test 58: missing llog files must not prevent MDT from mounting ========================================================== 19:16:53 (1788909413) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre total: 100 open/close in 1.36 seconds: 73.77 ops/second - unlinked 0 (time 1788909448 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec fail_loc=0 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 PASS 58 (78s) == conf-sanity test 59: writeconf mount option =========== 19:18:11 (1788909491) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy ln: failed to create symbolic link '/sbin/.libs': Read-only file system oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server original ost count: 1 (expect > 0) oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 after mdt writeconf count: 0 (expect 0) OST start without writeconf should fail: oleg339-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: No such file or directory oleg339-server: Is the MGS specification correct? oleg339-server: Is the filesystem name correct? oleg339-server: If upgrading, is the copied client log valid? (see upgrade docs) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory oleg339-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 OST start with writeconf should succeed: oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 after ost writeconf count: 1 (expect 1) oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 after ost2 writeconf count: 2 (expect 2) oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 59 (146s) == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 19:20:37 (1788909637) SKIP: conf-sanity test_60a ldiskfs only test SKIP 60a (1s) == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:20:38 (1788909638) SKIP: conf-sanity test_60b ldiskfs only test SKIP 60b (2s) == conf-sanity test 61a: large xattr ===================== 19:20:40 (1788909640) start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity grow value of trusted.big on /mnt/lustre/f61a.conf-sanity check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS Failing mds1 on oleg339-server Stopping /mnt/lustre-mds1 (opts:) on oleg339-server 19:21:07 (1788909667) shut down facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server Failover mds1 to oleg339-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 19:21:23 (1788909683) targets are mounted 19:21:23 (1788909683) facet_failover done oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 61a (76s) == conf-sanity test 61b: large xattr ===================== 19:21:56 (1788909716) SKIP: conf-sanity test_61b ldiskfs specific bug SKIP 61b (2s) == conf-sanity test 62: start with disabled journal ====== 19:21:58 (1788909718) SKIP: conf-sanity test_62 ldiskfs only test SKIP 62 (1s) SKIP: conf-sanity test_63 skipping excluded test 63 == conf-sanity test 64: check lfs df --lazy ============== 19:22:00 (1788909720) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 start ost2 service on oleg339-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre stop ost2 service on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server /home/green/git/lustre-release/lustre/utils/lfs df UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 2048 379520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 2048 378880 1% /mnt/lustre[OST:0] filesystem_summary: 382976 2048 378880 1% /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 64 (149s) == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 19:24:29 (1788909869) SKIP: conf-sanity test_65 ldiskfs only test SKIP 65 (1s) == conf-sanity test 66: replace nids ===================== 19:24:30 (1788909870) client=34682650 MDS=34682650 OSS=34682650 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Setting lustre-OST0000.osc.active from 1 to 0 Waiting 90s for '0' replace_nids should fail if MDS, OSTs and clients are UP oleg339-server: error: replace_nids: Operation now in progress pdsh@oleg339-client: oleg339-server: ssh exited with exit code 115 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) replace_nids should fail if MDS and OSTs are UP oleg339-server: error: replace_nids: Operation now in progress pdsh@oleg339-client: oleg339-server: ssh exited with exit code 115 stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server replace_nids should fail if MDS is UP oleg339-server: error: replace_nids: Operation now in progress pdsh@oleg339-client: oleg339-server: ssh exited with exit code 115 stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov -o nosvc lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Started lustre-MDT0000 command should accept two parameters pdsh@oleg339-client: oleg339-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] correct device name should be passed oleg339-server: error: replace_nids: No such device or address pdsh@oleg339-client: oleg339-server: ssh exited with exit code 6 wrong nids list should not destroy the system pdsh@oleg339-client: oleg339-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] pdsh@oleg339-client: oleg339-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] replace OST nid command should accept two parameters pdsh@oleg339-client: oleg339-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] wrong nids list should not destroy the system pdsh@oleg339-client: oleg339-server: ssh exited with exit code 4 replace primary NIDs for device (clients/servers must be unmounted) usage: replace_nids [,NID2,NID3:NID4,NID5:NID6] set NIDs with failover replace MDS nid stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Setting lustre-OST0000.osc.active from 0 to 1 Waiting 90s for '1' Updated after 4s: want '1' got '1' start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 66 (151s) SKIP: conf-sanity test_67 skipping excluded test 67 == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:27:02 (1788910022) umount lustre on /mnt/lustre..... start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 seq.ctl-lustre-MDT0000.fldb=[0x280000400-11811161088):0:mdt seq.srv-lustre-MDT0000.space=clear mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre seq.srv-lustre-MDT0000.space=[0x2c0000400 - 0x300000400]:0:mdt [0x200000bd1:0x1:0x0] umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 68 (32s) SKIP: conf-sanity test_69 skipping SLOW test 69 == conf-sanity test 70a: start MDT0, then OST, then MDT1 ========================================================== 19:27:34 (1788910054) SKIP: conf-sanity test_70a needs >= 2 MDTs SKIP 70a (1s) == conf-sanity test 70b: start OST, MDT1, MDT0 =========== 19:27:35 (1788910055) SKIP: conf-sanity test_70b needs >= 2 MDTs SKIP 70b (1s) == conf-sanity test 70c: stop MDT0, mkdir fail, create remote dir fail ========================================================== 19:27:37 (1788910057) SKIP: conf-sanity test_70c needs >= 2 MDTs SKIP 70c (2s) == conf-sanity test 70d: stop MDT1, mkdir succeed, create remote dir fail ========================================================== 19:27:38 (1788910058) SKIP: conf-sanity test_70d needs >= 2 MDTs SKIP 70d (1s) == conf-sanity test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 19:27:39 (1788910059) SKIP: conf-sanity test_70e needs >= 2 MDTs SKIP 70e (1s) == conf-sanity test 71a: start MDT0 OST0, MDT1, OST1 ===== 19:27:40 (1788910060) SKIP: conf-sanity test_71a needs >= 2 MDTs SKIP 71a (1s) == conf-sanity test 71b: start MDT1, OST0, MDT0, OST1 ==== 19:27:41 (1788910061) SKIP: conf-sanity test_71b needs >= 2 MDTs SKIP 71b (1s) == conf-sanity test 71c: start OST0, OST1, MDT1, MDT0 ==== 19:27:42 (1788910062) SKIP: conf-sanity test_71c needs >= 2 MDTs SKIP 71c (1s) == conf-sanity test 71d: start OST0, MDT1, MDT0, OST1 ==== 19:27:43 (1788910063) SKIP: conf-sanity test_71d needs >= 2 MDTs SKIP 71d (1s) == conf-sanity test 71e: start OST0, MDT1, OST1, MDT0 ==== 19:27:44 (1788910064) SKIP: conf-sanity test_71e needs >= 2 MDTs SKIP 71e (1s) == conf-sanity test 72: test fast symlink with extents flag enabled ========================================================== 19:27:45 (1788910065) SKIP: conf-sanity test_72 ldiskfs only test SKIP 72 (1s) == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:27:46 (1788910066) checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp autodegrade=on sys.timeout=20 failover.node=1.2.3.4@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.139@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:failover.node=1.2.3.4@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre failover_nids: [ "192.168.203.139@tcp", "1.2.3.4@tcp" ] umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server oleg339-server: oleg339-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg339-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 checking for existing Lustre data: not found stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 73a (40s) == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:28:27 (1788910107) checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp failover.node=1.2.3.4@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp failover.node=1.2.3.4@tcp autodegrade=on sys.timeout=20 autodegrade=onComment: just a comment for tail Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.139@tcp lustre:failover.node=1.2.3.4@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 lustre:userdata=just a comment for tail oleg339-server: tunefs.lustre: params are too long: oleg339-server: mgsnode=192.168.203.139@tcp:111.222.173.1@tcp,111.222.142.1@tcp:111.222.173.2@tcp,111.222.142.2@tcp:111.222.173.3@tcp,111.222.142.3@tcp:111.222.173.4@tcp,111.222.142.4@tcp:111.222.173.5@tcp,111.222.142.5@tcp:111.222.173.6@tcp,111.222.142.6@tcp:111.222.173.7@tcp,111.222.142.7@tcp:111.222.173.8@tcp,111.222.142.8@tcp:111.222.173.9@tcp,111.222.142.9@tcp:111.222.173.10@tcp,111.222.142.10@tcp:111.222.173.11@tcp,111.222.142.11@tcp:111.222.173.12@tcp,111.222.142.12@tcp:111.222.173.13@tcp,111.222.142.13@tcp:111.222.173.14@tcp,111.222.142.14@tcp:111.222.173.15@tcp,111.222.142.15@tcp:111.222.173.16@tcp,111.222.142.16@tcp:111.222.173.17@tcp,111.222.142.17@tcp:111.222.173.18@tcp,111.222.142.18@tcp:111.222.173.19@tcp,111.222.142.19@tcp:111.222.173.20@tcp,111.222.142.20@tcp:111.222.173.21@tcp,111.222.142.21@tcp:111.222.173.22@tcp,111.222.142.22@tcp:111.222.173.23@tcp,111.222.142.23@tcp:111.222.173.24@tcp,111.222.142.24@tcp:111.222.173.25@tcp,111.222.142.25@tcp:111.222.173.26@tcp,111.222.142.26@tcp:111.222.173.27@tcp,111.222.142.27@tcp:111.222.173.28@tcp,111.222.142.28@tcp:111.222.173.29@tcp,111.222.142.29@tcp:111.222.173.30@tcp,111.222.142.30@tcp:111.222.173.31@tcp,111.222.142.31@tcp:111.222.173.32@tcp,111.222.142.32@tcp:111.222.173.33@tcp,111.222.142.33@tcp:111.222.173.34@tcp,111.222.142.34@tcp:111.222.173.35@tcp,111.222.142.35@tcp:111.222.173.36@tcp,111.222.142.36@tcp:111.222.173.37@tcp,111.222.142.37@tcp:111.222.173.38@tcp,111.222.142.38@tcp:111.222.173.39@tcp,111.222.142.39@tcp:111.222.173.40@tcp,111.222.142.40@tcp:111.222.173.41@tcp,111.222.142.41@tcp:111.222.173.42@tcp,111.222.142.42@tcp:111.222.173.43@tcp,111.222.142.43@tcp:111.222.173.44@tcp,111.222.142.44@tcp:111.222.173.45@tcp,111.222.142.45@tcp:111.222.173.46@tcp,111.222.142.46@tcp:111.222.173.47@tcp,111.222.142.47@tcp:111.222.173.48@tcp,111.222.142.48@tcp:111.222.173.49@tcp,111.222.142.49@tcp failover.node=1.2.3.4@tcp:111.222.173.1@tcp,111.222.142.1@tcp:111.222.173.2@tcp,111.222.142.2@tcp:111.222.173.3@tcp,111.222.142.3@tcp:111.222.173.4@tcp,111.222.142.4@tcp:111.222.173.5@tcp,111.222.142.5@tcp:111.222.173.6@tcp,111.222.142.6@tcp:111.222.173.7@tcp,111.222.142.7@tcp:111.222.173.8@tcp,111.222.142.8@tcp:111.222.173.9@tcp,111.222.142.9@tcp:111.222.173.10@tcp,111.222.142.10@tcp:111.222.173.11@tcp,111.222.142.11@tcp:111.222.173.12@tcp,111.222.142.12@tcp:111.222.173.13@tcp,111.222.142.13@tcp:111.222.173.14@tcp,111.222.142.14@tcp:111.222.173.15@tcp,111.222.142.15@tcp:111.222.173.16@tcp,111.222.142.16@tcp:111.222.173.17@tcp,111.222.142.17@tcp:111.222.173.18@tcp,111.222.142.18@tcp:111.222.173.19@tcp,111.222.142.19@tcp:111.222.173.20@tcp,111.222.142.20@tcp:111.222.173.21@tcp,111.222.142.21@tcp:111.222.173.22@tcp,111.222.142.22@tcp:111.222.173.23@tcp,111.222.142.23@tcp:111.222.173.24@tcp,111.222.142.24@tcp:111.222.173.25@tcp,111.222.142.25@tcp:111.222.173.26@tcp,111.222.142.26@tcp:111.222.173.27@tcp,111.222.142.27@tcp:111.222.173.28@tcp,111.222.142.28@tcp:111.222.173.29@tcp,111.222.142.29@tcp:111.222.173.30@tcp,111.222.142.30@tcp:111.222.173.31@tcp,111.222.142.31@tcp:111.222.173.32@tcp,111.222.142.32@tcp:111.222.173.33@tcp,111.222.142.33@tcp:111.222.173.34@tcp,111.222.142.34@tcp:111.222.173.35@tcp,111.222.142.35@tcp:111.222.173.36@tcp,111.222.142.36@tcp:111.222.173.37@tcp,111.222.142.37@tcp:111.222.173.38@tcp,111.222.142.38@tcp:111.222.173.39@tcp,111.222.142.39@tcp:111.222.173.40@tcp,111.222.142.40@tcp:111.222.173.41@tcp,111.222.142.41@tcp:111.222.173.42@tcp,111.222.142.42@tcp:111.222.173.43@tcp,111.222.142.43@tcp:111.222.173.44@tcp,111.222.142.44@tcp:111.222.173.45@tcp,111.222.142.45@tcp:111.222.173.46@tcp,111.222.142.46@tcp:111.222.173.47@tcp,111.222.142.47@tcp:111.222.173.48@tcp,111.222.142.48@tcp:111.222.173.49@tcp,111.222.142.49@tcp:111.222.173.50@tcp,111.222.142.50@tcp autodegrade=on sys.timeout=20 +mgsnode==111.222.173.50@tcp,111.222.142.50@tcp oleg339-server: tunefs.lustre: exiting with 7 (Argument list too long) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 7 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Found total 198 NIDs in parameters Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 PASS 73b (63s) == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 19:29:29 (1788910169) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory oleg339-server: sh: /usr/sbin/ksocklnd-config: No such file or directory restart with 35 nets start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. PASS 73c (269s) == conf-sanity test 73d: erase + new parameter doesn't corrupt mountdata ========================================================== 19:33:58 (1788910438) checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=1 autodegrade=on Writing lustre-ost1/ost1 properties lustre:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=1 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=1 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=10.23.7.1@tcp:10.23.7.2@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:mgsnode=10.23.7.1@tcp:10.23.7.2@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 PASS 73d (33s) == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 19:34:31 (1788910471) umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server stop mds service on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing lnet_if_list Node: oleg339-server, rinf=ens2 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 oleg339-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.139 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp50 local NI(s): - nid: 192.168.203.39@tcp50 status: up interfaces: 0: ens2 Try to mount to MGS NID 192.168.203.139@tcp50 stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.39 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp50 local NI(s): - nid: 192.168.203.39@tcp50 status: up interfaces: 0: ens2 Try to mount after MGS remount setup single mount lustre success UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 2048 379520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 2048 378880 1% /mnt/lustre[OST:0] filesystem_summary: 382976 2048 378880 1% /mnt/lustre umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. PASS 73e (170s) == conf-sanity test 73f: mgc register before connect ===== 19:37:21 (1788910641) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 fail_loc=0x8000090f start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 umount lustre on /mnt/lustre..... stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. PASS 73f (61s) == conf-sanity test 74: Test per-device adaptive timeout parameters ========================================================== 19:38:22 (1788910702) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Permanent disk data: Device: lustre-mdt1_2/mdt1_2 Target: fs15246:MDT0000 Index: 0 Lustre FS: fs15246 Mount type: zfs Flags: 0x61 (MDT first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1_2/mdt1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=97 lustre:index=0 lustre:fsname=fs15246 lustre:svname=fs15246:MDT0000 Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: fs15246:OST0000 Index: 0 Lustre FS: fs15246 Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=fs15246 lustre:svname=fs15246:OST0000 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1849: OST_INDICES: bad array subscript Start fs2ost: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-fs15246-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started fs15246-OST0000 Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1_2/mdt1_2 Started fs15246-MDT0000 ========== All mounted lustre fs ==================== 192.168.203.139@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) 192.168.203.139@tcp:/fs15246 on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,nouser_xattr,verbose) ===================================================== osc.lustre-OST0000-osc-ffff9a4903494000.ldlm_enqueue_min=99 obdfilter.lustre-OST0000.at_min=1 mdc.lustre-MDT0000-mdc-ffff9a4903494000.at_max=599 mdt.lustre-MDT0000.at_max=599 obdfilter.lustre-OST0000.at_max=599 mgc.MGC192.168.203.139@tcp.at_history=588 mgs.MGS.at_history=588 mgs.MGS.at_history=0 mgc.MGC192.168.203.139@tcp.at_history=0 mdt.lustre-MDT0000.at_max=0 obdfilter.lustre-OST0000.at_max=0 mdc.lustre-MDT0000-mdc-ffff9a4903494000.at_max=0 obdfilter.lustre-OST0000.at_min=0 osc.lustre-OST0000-osc-ffff9a4903494000.ldlm_enqueue_min=0 umount /mnt/lustre2 ... stopping fs2mds ... Stopping /mnt/lustre-fs2mds (opts:-f) on oleg339-server stopping fs2ost ... Stopping /mnt/lustre-fs2ost (opts:-f) on oleg339-server umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. PASS 74 (125s) == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 19:40:27 (1788910827) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Permanent disk data: Device: lustre-mdt1/mdt1 Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1 /dev/vdc mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1/mdt1 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 Permanent disk data: Device: lustre-mdt1/mdt1 Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1 /dev/vdc mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1/mdt1 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=69 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server PASS 75 (119s) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 7690 sec ========== 19:43:03 (1788910983) === conf-sanity: start cleanup 19:43:04 (1788910984) === === conf-sanity: finish cleanup 19:43:06 (1788910986) ===