-----============= acceptance-small: conf-sanity ============----- Mon Mar 16 09:37:29 EDT 2026
mgs: Rocky Linux release 8.10 (Green Obsidian)
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
mds1: Rocky Linux release 8.10 (Green Obsidian)
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
ost1: Rocky Linux release 8.10 (Green Obsidian)
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
client: Rocky Linux release 8.10 (Green Obsidian)
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg408-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory
excepting tests: 21e 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a
skipping tests SLOW=no: 45 69 106 111 114
Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg408-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server
oleg408-server: oleg408-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

== conf-sanity test 0: single mount setup ================ 09:38:44 (1773668324)
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 0 (31s)

== conf-sanity test 1: start up ost twice (should return errors) ========================================================== 09:39:15 (1773668355)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost second time...
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg408-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of lustre-ost1/ost1 on ost1 failed 17
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 1 (40s)

== conf-sanity test 2: start up mds twice (should return err) ========================================================== 09:39:55 (1773668395)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start mds second time..
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17
Start of lustre-mdt1/mdt1 on mds1 failed 17
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
PASS 2 (47s)

== conf-sanity test 3: mount client twice (should return err) ========================================================== 09:40:42 (1773668442)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
mount.lustre: according to /etc/mtab 192.168.204.108@tcp:/lustre is already mounted on /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 3 (44s)

== conf-sanity test 4: force cleanup ost, then cleanup === 09:41:26 (1773668486)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
PASS 4 (74s)

== conf-sanity test 5a: force cleanup mds, then cleanup == 09:42:40 (1773668560)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
                     USER        PID ACCESS COMMAND
/mnt/lustre:         root     kernel mount /mnt/lustre
/mnt/lustre is in use by user space process.
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server
killing umount
waiting for umount to finish
/home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 149: 27227 Terminated              $UMOUNT -f $MOUNT
manual umount lustre on /mnt/lustre....
umount: /mnt/lustre: not mounted.
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
manual umount lustre on /mnt/lustre....
umount: /mnt/lustre: not mounted.
/etc/mtab updated in 0 secs
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 5a (94s)

== conf-sanity test 5b: Try to start a client with no MGS (should return errs) ========================================================== 09:44:14 (1773668654)
start ost1 service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Connection timed out
umount lustre on /mnt/lustre.....
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 5b (250s)

== conf-sanity test 5c: cleanup after failed mount (bug 2712) (should return errs) ========================================================== 09:48:24 (1773668904)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount wrong.lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/wrong.lustre /mnt/lustre
mount.lustre: mount 192.168.204.108@tcp:/wrong.lustre at /mnt/lustre failed: File name too long
umount lustre on /mnt/lustre.....
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
PASS 5c (51s)

== conf-sanity test 5d: mount with ost down ============== 09:49:15 (1773668955)
start ost1 service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 5d (117s)

== conf-sanity test 5e: delayed connect, don't crash (bug 10268) ========================================================== 09:51:12 (1773669072)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
fail_loc=0x80000506
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
PASS 
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 5e (66s)

== conf-sanity test 5f: mds down, cleanup after failed mount (bug 2712) ========================================================== 09:52:18 (1773669138)

 SKIP: conf-sanity test_5f needs separate mgs and mds
SKIP 5f (2s)

== conf-sanity test 5g: handle missing debugfs =========== 09:52:20 (1773669140)
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
none /sys/kernel/debug debugfs rw,relatime 0 0
PASS 5g (15s)

== conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:52:35 (1773669155)
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
Stopping /mnt/lustre-mds1 (opts:) on oleg408-server
fail_loc=0x80000135
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: No such file or directory
oleg408-server: Is the MGS specification correct?
oleg408-server: Is the filesystem name correct?
oleg408-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
Start of lustre-mdt1/mdt1 on mds1 failed 2
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 5h (100s)

== conf-sanity test 5i: start mdt failure at mdt_quota_init() ========================================================== 09:54:15 (1773669255)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
Stopping /mnt/lustre-mds1 (opts:) on oleg408-server
fail_loc=0x80000A05
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: Bad file descriptor
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 9
Start of lustre-mdt1/mdt1 on mds1 failed 9
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 5i (117s)

== conf-sanity test 5j: verify libcfs doesn't crash when setting debug flags ========================================================== 09:56:13 (1773669373)

LNET unconfigure error 22: Invalid argument
unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
oleg408-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
debug=all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 5j (14s)

== conf-sanity test 6: manual umount, then mount again === 09:56:26 (1773669386)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
manual umount lustre on /mnt/lustre....
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 6 (84s)

== conf-sanity test 7: manual umount, then cleanup ======= 09:57:50 (1773669470)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
manual umount lustre on /mnt/lustre....
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 7 (92s)

== conf-sanity test 8: double mount setup ================ 09:59:22 (1773669562)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre2.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2
setup double mount lustre success
umount lustre on /mnt/lustre2.....
Stopping client oleg408-client.virtnet /mnt/lustre2 (opts:)
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 8 (79s)

== conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================================== 10:00:41 (1773669641)
start ost1 service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
debug=inode trace
subsystem_debug=mds ost
lnet.debug success
lnet.subsystem_debug success
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
PASS 9 (80s)

== conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:02:01 (1773669721)
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
PASS 10a (45s)

== conf-sanity test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 10:02:46 (1773669766)

 SKIP: conf-sanity test_17 ldiskfs only test
SKIP 17 (2s)

== conf-sanity test 18: check mkfs creates large journals ========================================================== 10:02:48 (1773669768)

 SKIP: conf-sanity test_18 ldiskfs only test
SKIP 18 (1s)

== conf-sanity test 19a: start/stop MDS without OSTs ===== 10:02:49 (1773669769)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server
PASS 19a (26s)

== conf-sanity test 19b: start/stop OSTs without MDS ===== 10:03:15 (1773669795)
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
PASS 19b (56s)

== conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:04:11 (1773669851)
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
setup single mount lustre success
mount lustre with opts remount,ro on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o remount,ro 192.168.204.108@tcp:/lustre /mnt/lustre
touch: cannot touch '/mnt/lustre/f20.conf-sanity': Read-only file system
mount lustre with opts remount,rw on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o remount,rw 192.168.204.108@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
PASS 20 (26s)

== conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:04:37 (1773669877)
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server
PASS 21a (24s)

== conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:05:02 (1773669902)
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
PASS 21b (81s)

== conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:06:22 (1773669982)
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost2 service on oleg408-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg408-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop ost2 service on oleg408-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server
oleg408-server: 
oleg408-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg408-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg408-server: 
oleg408-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg408-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg408-server: 
oleg408-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg408-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs.lustre failed, reformatting instead
Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f)
oleg408-server: oleg408-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server
PASS 21c (134s)

== conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:08:36 (1773670116)

 SKIP: conf-sanity test_21d need separate mgs device
SKIP 21d (1s)

 SKIP: conf-sanity test_21e skipping excluded test 21e

== conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:08:38 (1773670118)
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Client mount with ost in logs, but none running
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:-f)
PASS 
Client mount with a running ost
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg408-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff949484c0a000.ost_server_uuid 50
osc.lustre-OST0000-osc-ffff949484c0a000.ost_server_uuid in FULL state after 0 sec
setup single mount lustre success
PASS 
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 22 (54s)

== conf-sanity test 23a: interrupt client during recovery mount delay ========================================================== 10:09:32 (1773670172)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
Stopping /mnt/lustre-mds1 (opts:) on oleg408-server
Stopping client /mnt/lustre (opts: -f)
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
mount pid is 79821, mount.lustre pid is 79822
    PID TTY          TIME CMD
  79822 pts/0    00:00:00 mount.lustre
    PID TTY          TIME CMD
waiting for mount to finish
root       79821   79820  0 10:10 pts/0    00:00:00 mount -t lustre -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
root       79822   79821  0 10:10 pts/0    00:00:00 /sbin/mount.lustre 192.168.204.108@tcp:/lustre /mnt/lustre -o rw,user_xattr,flock
root       79859   77668  0 10:10 pts/0    00:00:00 grep mount
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6872: 79821 Terminated              LUSTRE="/home/green/git/lustre-release/lustre" bash -c "mount -t lustre -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre"
PID1=
PID2=
umount lustre on /mnt/lustre.....
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 23a (66s)

== conf-sanity test 23b: Simulate -EINTR during mount ==== 10:10:38 (1773670238)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
fail_loc=0x80000313
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
PASS 23b (41s)

== conf-sanity test 24a: Multiple MDTs on a single node == 10:11:19 (1773670279)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1

   Permanent disk data:
Target:     969362ae:MDT0000
Index:      0
Lustre FS:  969362ae
Mount type: zfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.108@tcp:192.168.204.108@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1_2/mdt1_2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1_2/mdt1_2 properties
  lustre:mgsnode=192.168.204.108@tcp:192.168.204.108@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=97
  lustre:index=0
  lustre:fsname=969362ae
  lustre:svname=969362ae:MDT0000

   Permanent disk data:
Target:     969362ae:OST0000
Index:      0
Lustre FS:  969362ae
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1_2/ost1_2
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1_2/ost1_2 properties
  lustre:mgsnode=192.168.204.108@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=969362ae
  lustre:svname=969362ae:OST0000
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
Start fs2mds: mount -t lustre -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1_2/mdt1_2
Started 969362ae-MDT0000
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1837: OST_INDICES: bad array subscript
Start fs2ost: mount -t lustre -o localrecov  lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost
seq.cli-969362ae-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Commit the device label on lustre-ost1_2/ost1_2
Started 969362ae-OST0000
oleg408-server: fstrim: /mnt/lustre-fs2ost: the discard operation is not supported
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
969362ae-MDT0000: Deregistered changelog user #1
setup single mount lustre success
Failing fs2mds on oleg408-server
Stopping /mnt/lustre-fs2mds (opts:) on oleg408-server
10:12:27 (1773670347) shut down
facet: fs2mds facet_host: oleg408-server facet_failover_host: 
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4320: [: oleg408-server: unary operator expected
mount facets: fs2mds
Start fs2mds: mount -t lustre -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started 969362ae-MDT0000
10:12:41 (1773670361) targets are mounted
10:12:41 (1773670361) facet_failover done
Failing fs2ost on oleg408-server
Stopping /mnt/lustre-fs2ost (opts:) on oleg408-server
10:12:42 (1773670362) shut down
facet: fs2ost facet_host: oleg408-server facet_failover_host: 
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4320: [: oleg408-server: unary operator expected
mount facets: fs2ost
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1837: OST_INDICES: bad array subscript
Start fs2ost: mount -t lustre -o localrecov  lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost
seq.cli-969362ae-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started 969362ae-OST0000
10:12:57 (1773670377) targets are mounted
10:12:57 (1773670377) facet_failover done
Filesystem                                                                                                         1K-blocks        Used   Available Use% Mounted on
devtmpfs                                                                                                             1890908           0     1890908   0% /dev
tmpfs                                                                                                                1890908           0     1890908   0% /dev/shm
tmpfs                                                                                                                1890908       16356     1874552   1% /run
tmpfs                                                                                                                1890908           0     1890908   0% /sys/fs/cgroup
/dev/nbd0                                                                                                           30787492    17460884    11737360  60% /
none                                                                                                                 1890908       24372     1866536   2% /var/lib/stateless/writable
/dev/vda                                                                                                               67328       67328           0 100% /home/green/git/lustre-release
192.168.200.253:/exports/state/oleg408-client.virtnet                                                              143911936   133210112     4058112  98% /var/lib/stateless/state
192.168.200.253:/exports/state/oleg408-client.virtnet/boot                                                         143911936   133210112     4058112  98% /boot
192.168.200.253:/exports/state/oleg408-client.virtnet/etc/kdump.conf                                               143911936   133210112     4058112  98% /etc/kdump.conf
tmpfs                                                                                                                 378180           0      378180   0% /run/user/0
192.168.200.253:/exports/testreports/61262/testresults/conf-sanity1-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry2 54445813760 42126869504 12318944256  78% /tmp/testlogs
192.168.204.108@tcp:/lustre                                                                                           382976        3072      377856   1% /mnt/lustre
192.168.204.108@tcp:/969362ae                                                                                         244736        3072      239616   2% /mnt/lustre2
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server
umount /mnt/lustre2 ...
stopping fs2mds ...
Stopping /mnt/lustre-fs2mds (opts:-f) on oleg408-server
stopping fs2ost ...
Stopping /mnt/lustre-fs2ost (opts:-f) on oleg408-server
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 24a (120s)

== conf-sanity test 24b: Multiple MGSs on a single node (should return err) ========================================================== 10:13:19 (1773670399)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1

   Permanent disk data:
Target:     lustre2:MDT0000
Index:      0
Lustre FS:  lustre2
Mount type: zfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1_2/mdt1_2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1_2/mdt1_2 properties
  lustre:mgsnode=192.168.204.108@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=101
  lustre:index=0
  lustre:fsname=lustre2
  lustre:svname=lustre2:MDT0000
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
Start fs2mds: mount -t lustre -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg408-server: mount.lustre: mount lustre-mdt1_2/mdt1_2 at /mnt/lustre-fs2mds failed: Operation already in progress
oleg408-server: The target service is already running. (lustre-mdt1_2/mdt1_2)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 114
Start of lustre-mdt1_2/mdt1_2 on fs2mds failed 114
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 24b (47s)

== conf-sanity test 25: Verify modules are referenced ==== 10:14:07 (1773670447)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
PASS 25 (44s)

== conf-sanity test 26: MDT startup failure cleans LOV (should return errs) ========================================================== 10:14:50 (1773670490)
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
fail_loc=0x80000135
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: No such file or directory
oleg408-server: Is the MGS specification correct?
oleg408-server: Is the filesystem name correct?
oleg408-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
Start of lustre-mdt1/mdt1 on mds1 failed 2
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 26 (26s)

== conf-sanity test 27a: Reacquire MGS lock if OST started first ========================================================== 10:15:16 (1773670516)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg408-server
stop mds service on oleg408-server

LNET unconfigure error 22: Invalid argument
unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
oleg408-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
start ost1 service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Requeue thread should have started: 
Setting lustre-OST0000.ost.client_cache_seconds from 110 to 115
Waiting 90s for '115'
Updated after 5s: want '115' got '115'
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
PASS 27a (88s)

== conf-sanity test 27b: Reacquire MGS lock after failover ========================================================== 10:16:44 (1773670604)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
Failing mds1 on oleg408-server
Stopping /mnt/lustre-mds1 (opts:) on oleg408-server
10:17:13 (1773670633) shut down
facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server
Failover mds1 to oleg408-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
10:17:27 (1773670647) targets are mounted
10:17:27 (1773670647) facet_failover done
Setting lustre-MDT0000.mdt.identity_acquire_expire from 30 to 35
Waiting 90s for '35'
Updated after 4s: want '35' got '35'
Setting lustre-MDT0000.mdc.max_rpcs_in_flight from 8 to 13
Waiting 90s for '13'
Updated after 4s: want '13' got '13'
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 27b (63s)

== conf-sanity test 28A: permanent parameter setting ===== 10:17:47 (1773670667)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
ORIG:4 MAX:64
ORIG:4 MAX:64
Setting lustre.llite.max_read_ahead_whole_mb from 4 to 5
Waiting 90s for '5'
Setting lustre.llite.max_read_ahead_whole_mb from 5 to 6
Waiting 90s for '6'
Updated after 7s: want '6' got '6'
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
New config success: got 6
Setting lustre.llite.max_read_ahead_whole_mb from 6 to 4
Waiting 90s for '4'
Updated after 7s: want '4' got '4'
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 28A (56s)

== conf-sanity test 28a: set symlink parameters permanently with lctl ========================================================== 10:18:43 (1773670723)

 SKIP: conf-sanity test_28a LU-4221: no such proc params for ZFS OSTs
SKIP 28a (1s)

== conf-sanity test 28b: verify client-side parameters are set upon mount ========================================================== 10:18:44 (1773670724)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
jobid_name=test_28b-25750%p
before mount: %e.%u
after mount:  test_28b-25750%p
jobid_name=%e.%u
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
PASS 28b (42s)

== conf-sanity test 28c: verify filesystem parameters are set upon mount ========================================================== 10:19:26 (1773670766)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
jobid_name=test_28c-3530%p
before mount: %e.%u
after mount:  test_28c-3530%p
jobid_name=%e.%u
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
PASS 28c (43s)

== conf-sanity test 29: permanently remove an OST ======== 10:20:09 (1773670809)
start ost2 service on oleg408-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Setting lustre-OST0001.osc.active from 1 to 0
Waiting 90s for '0'
Updated after 10s: want '0' got '0'
Live client success: got lustre-OST0001_UUID	IDLE	DEACTIVATED
osc.lustre-OST0001-osc-MDT0000.active updated after 0 sec (got 0)
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
New client success: got 'lustre-OST0001_UUID	NEW	DEACTIVATED'
Setting lustre-OST0001.osc.active from 0 to 1
Waiting 90s for '1'
Updated after 9s: want '1' got '1'
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg408-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
oleg408-server: 
oleg408-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg408-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg408-server: 
oleg408-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg408-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg408-server: 
oleg408-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg408-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs.lustre failed, reformatting instead
Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
oleg408-server: oleg408-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server
PASS 29 (138s)

== conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 10:22:27 (1773670947)
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
Big config llog
Setting lustre.llite.max_read_ahead_whole_mb from 4 to 1
Waiting 90s for '1'
Setting lustre.llite.max_read_ahead_whole_mb from 1 to 2
Waiting 90s for '2'
Updated after 8s: want '2' got '2'
Setting lustre.llite.max_read_ahead_whole_mb from 2 to 3
Waiting 90s for '3'
Waiting 80s for '3'
Updated after 11s: want '3' got '3'
Setting lustre.llite.max_read_ahead_whole_mb from 3 to 4
Waiting 90s for '4'
Updated after 8s: want '4' got '4'
Setting lustre.llite.max_read_ahead_whole_mb from 4 to 5
Waiting 90s for '5'
Updated after 8s: want '5' got '5'
Setting lustre.llite.max_read_ahead_whole_mb from 5 to 4
Waiting 90s for '4'
Updated after 7s: want '4' got '4'
Setting lustre.llite.max_read_ahead_whole_mb from 4 to 3
Waiting 90s for '3'
Updated after 7s: want '3' got '3'
Setting lustre.llite.max_read_ahead_whole_mb from 3 to 2
Waiting 90s for '2'
Updated after 10s: want '2' got '2'
Setting lustre.llite.max_read_ahead_whole_mb from 2 to 1
Waiting 90s for '1'
Updated after 10s: want '1' got '1'
Setting lustre.llite.max_read_ahead_whole_mb from 1 to 2
Waiting 90s for '2'
Updated after 5s: want '2' got '2'
Setting lustre.llite.max_read_ahead_whole_mb from 2 to 3
Waiting 90s for '3'
Updated after 8s: want '3' got '3'
Setting lustre.llite.max_read_ahead_whole_mb from 3 to 4
Waiting 90s for '4'
Updated after 5s: want '4' got '4'
Setting lustre.llite.max_read_ahead_whole_mb from 4 to 5
Waiting 90s for '5'
Updated after 9s: want '5' got '5'
Setting lustre.llite.max_read_ahead_whole_mb from 5 to 4
Waiting 90s for '4'
Updated after 8s: want '4' got '4'
Setting lustre.llite.max_read_ahead_whole_mb from 4 to 3
Waiting 90s for '3'
Updated after 7s: want '3' got '3'
Setting lustre.llite.max_read_ahead_whole_mb from 3 to 2
Waiting 90s for '2'
Updated after 6s: want '2' got '2'
Setting lustre.llite.max_read_ahead_whole_mb from 2 to 1
Waiting 90s for '1'
Updated after 6s: want '1' got '1'
Setting lustre.llite.max_read_ahead_whole_mb from 1 to 2
Waiting 90s for '2'
Updated after 10s: want '2' got '2'
Setting lustre.llite.max_read_ahead_whole_mb from 2 to 3
Waiting 90s for '3'
Updated after 7s: want '3' got '3'
Setting lustre.llite.max_read_ahead_whole_mb from 3 to 4
Waiting 90s for '4'
Updated after 6s: want '4' got '4'
Setting lustre.llite.max_read_ahead_whole_mb from 4 to 5
Waiting 90s for '5'
Updated after 8s: want '5' got '5'
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
PASS 
Erase parameter setting
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
deleted (default) value=4, orig=4
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 30a (191s)

== conf-sanity test 30b: Remove failover nids ============ 10:25:38 (1773671138)
start mds service on oleg408-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1
Started lustre-OST0000
oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
Checking peer and local NIDs:
192.168.204.108@tcp
Changing 192.168.204.108@tcp to fake NID 192.168.204.89@tcp
Setting lustre-OST0000.failover.node from 192.168.204.108@tcp to 192.168.204.89@tcp
Waiting 90s for '192.168.204.89@tcp'
should have 5 failover NIDs, have 5
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
only 4 failover NIDs should be left, have 4
umount lustre on /mnt/lustre.....
Stopping client oleg408-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg408-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server
stop mds service on oleg408-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
PASS 30b (42s)

== conf-sanity test 31: Connect to non-existent node (shouldn't crash) ========================================================== 10:26:20 (1773671180)
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
mount.lustre: mount 4.3.2.1@tcp:/lustre at /mnt/lustre failed: Connection timed out
umount lustre on /mnt/lustre.....
stop ost1 service on oleg408-server
stop mds service on oleg408-server

unloading modules via unload_modules_local on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing unload_modules_local
oleg408-server: 
modules unloaded.
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2
PASS 31 (282s)
Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
oleg408-server: oleg408-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg408-server'
oleg408-server: oleg408-server.virtnet: executing load_modules_local
oleg408-server: Loading modules from /home/green/git/lustre-release/lustre
oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg408-server: detected 4 online CPUs by sysfs
oleg408-server: MODOPTS_LIBCFS=
oleg408-server: Force libcfs to create 2 CPU partitions
oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
== conf-sanity test complete, duration 3243 sec ========== 10:31:32 (1773671492)
=== conf-sanity: start cleanup 10:31:33 (1773671493) ===
=== conf-sanity: finish cleanup 10:31:34 (1773671494) ===