-----============= acceptance-small: conf-sanity ============----- Tue Jun 23 10:28:35 EDT 2026
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg211-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex
- see CLIENT_VERSION > v2_15_63-53-g1a6ef725c2 (34682451 > 34553653) for LU-16938, go 81
excepting tests: 21e 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a
skipping tests SLOW=no: 45 69 106 111 114
Stopping clients: oleg211-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg211-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg211-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg211-server
oleg211-server: oleg211-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg211-server: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg211-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

== conf-sanity test 0: single mount setup ================ 10:31:17 (1782225077)
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 0 (87s)

== conf-sanity test 1: start up ost twice (should return errors) ========================================================== 10:32:44 (1782225164)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost second time...
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg211-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of lustre-ost1/ost1 on ost1 failed 17
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 1 (113s)

== conf-sanity test 2: start up mds twice (should return err) ========================================================== 10:34:37 (1782225277)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start mds second time..
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 17
Start of lustre-mdt1/mdt1 on mds1 failed 17
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 2 (107s)

== conf-sanity test 3: mount client twice (should return err) ========================================================== 10:36:24 (1782225384)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
mount.lustre: according to /etc/mtab 192.168.202.111@tcp:/lustre is already mounted on /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
PASS 3 (101s)

== conf-sanity test 4: force cleanup ost, then cleanup === 10:38:05 (1782225485)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 4 (120s)

== conf-sanity test 5a: force cleanup mds, then cleanup == 10:40:05 (1782225605)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
                     USER        PID ACCESS COMMAND
/mnt/lustre:         root     kernel mount /mnt/lustre
/mnt/lustre is in use by user space process.
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server
killing umount
waiting for umount to finish
/home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 149: 27617 Terminated              $UMOUNT -f $MOUNT
manual umount lustre on /mnt/lustre....
umount: /mnt/lustre: not mounted.
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
manual umount lustre on /mnt/lustre....
umount: /mnt/lustre: not mounted.
/etc/mtab updated in 0 secs
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
PASS 5a (127s)

== conf-sanity test 5b: Try to start a client with no MGS (should return errs) ========================================================== 10:42:12 (1782225732)
start ost1 service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
mount.lustre: mount 192.168.202.111@tcp:/lustre at /mnt/lustre failed: Connection timed out
umount lustre on /mnt/lustre.....
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 5b (285s)

== conf-sanity test 5c: cleanup after failed mount (bug 2712) (should return errs) ========================================================== 10:46:57 (1782226017)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount wrong.lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/wrong.lustre /mnt/lustre
mount.lustre: mount 192.168.202.111@tcp:/wrong.lustre at /mnt/lustre failed: File name too long
umount lustre on /mnt/lustre.....
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 5c (104s)

== conf-sanity test 5d: mount with ost down ============== 10:48:41 (1782226121)
start ost1 service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 5d (169s)

== conf-sanity test 5e: delayed connect, don't crash (bug 10268) ========================================================== 10:51:30 (1782226290)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
fail_loc=0x80000506
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
PASS 
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
PASS 5e (118s)

== conf-sanity test 5f: mds down, cleanup after failed mount (bug 2712) ========================================================== 10:53:29 (1782226409)

 SKIP: conf-sanity test_5f needs separate mgs and mds
SKIP 5f (3s)

== conf-sanity test 5g: handle missing debugfs =========== 10:53:32 (1782226412)
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
none /sys/kernel/debug debugfs rw,relatime 0 0
PASS 5g (30s)

== conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:54:02 (1782226442)
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
Stopping /mnt/lustre-mds1 (opts:) on oleg211-server
fail_loc=0x80000135
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: No such file or directory
oleg211-server: Is the MGS specification correct?
oleg211-server: Is the filesystem name correct?
oleg211-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
Start of lustre-mdt1/mdt1 on mds1 failed 2
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 5h (115s)

== conf-sanity test 5i: start mdt failure at mdt_quota_init() ========================================================== 10:55:57 (1782226557)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
Stopping /mnt/lustre-mds1 (opts:) on oleg211-server
fail_loc=0x80000A05
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: Bad file descriptor
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 9
Start of lustre-mdt1/mdt1 on mds1 failed 9
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
PASS 5i (146s)

== conf-sanity test 5j: verify libcfs doesn't crash when setting debug flags ========================================================== 10:58:23 (1782226703)

LNET unconfigure error 22: Invalid argument
unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
oleg211-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
debug=all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 5j (14s)

== conf-sanity test 6: manual umount, then mount again === 10:58:38 (1782226718)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
manual umount lustre on /mnt/lustre....
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 6 (122s)

== conf-sanity test 7: manual umount, then cleanup ======= 11:00:41 (1782226841)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
manual umount lustre on /mnt/lustre....
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
PASS 7 (118s)

== conf-sanity test 8: double mount setup ================ 11:02:38 (1782226958)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre2.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre2
setup double mount lustre success
umount lustre on /mnt/lustre2.....
Stopping client oleg211-client.virtnet /mnt/lustre2 (opts:)
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 8 (141s)

== conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================================== 11:04:59 (1782227099)
start ost1 service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
debug=inode trace
subsystem_debug=mds ost
lnet.debug success
lnet.subsystem_debug success
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
PASS 9 (116s)

== conf-sanity test 10a: find lctl param broken symlinks ========================================================== 11:06:56 (1782227216)
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2
PASS 10a (93s)

== conf-sanity test 11: Verify tool --version option works properly ========================================================== 11:08:30 (1782227310)
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2
PASS 11 (18s)

== conf-sanity test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 11:08:48 (1782227328)

 SKIP: conf-sanity test_17 ldiskfs only test
SKIP 17 (3s)

== conf-sanity test 18: check mkfs creates large journals ========================================================== 11:08:52 (1782227332)

 SKIP: conf-sanity test_18 ldiskfs only test
SKIP 18 (3s)

== conf-sanity test 19a: start/stop MDS without OSTs ===== 11:08:55 (1782227335)
start mds service on oleg211-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server
PASS 19a (56s)

== conf-sanity test 19b: start/stop OSTs without MDS ===== 11:09:51 (1782227391)
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
PASS 19b (78s)

== conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 11:11:09 (1782227469)
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
setup single mount lustre success
mount lustre with opts remount,ro on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o remount,ro 192.168.202.111@tcp:/lustre /mnt/lustre
touch: cannot touch '/mnt/lustre/f20.conf-sanity': Read-only file system
mount lustre with opts remount,rw on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o remount,rw 192.168.202.111@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
PASS 20 (78s)

== conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 11:12:27 (1782227547)
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg211-server: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg211-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server
PASS 21a (68s)

== conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 11:13:35 (1782227615)
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg211-server: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg211-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
PASS 21b (116s)

== conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 11:15:31 (1782227731)
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost2 service on oleg211-server