-----============= acceptance-small: conf-sanity ============----- Tue Sep  8 07:05:58 EDT 2026
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg224-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex
- see CLIENT_VERSION >= v2_15_63-53-g1a6ef725c2 (34683431 >= 34553653) for LU-16938, go 81
- see CLIENT_VERSION >= v2_15_65-31-g74a5b9efaa (34683431 >= 34554143) for LU-18348, go 32b 32c
- see CLIENT_VERSION >= v2_16_55-19-gf69f27cc35 (34683431 >= 34617107) for LU-17920, go 123ae 123ag
- see CLIENT_VERSION >= v2_17_52-94-g86904ef943 (34683431 >= 34681950) for LU-19740, go 115
excepting tests: 21e 32b 32c 32newtarball 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a
skipping tests SLOW=no: 45 69 106 111 114
Stopping clients: oleg224-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg224-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg224-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg224-server
oleg224-server: oleg224-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server

== conf-sanity test 0: single mount setup ================ 07:09:22 (1788865762)
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
PASS 0 (91s)

== conf-sanity test 1: start up ost twice (should return errors) ========================================================== 07:10:53 (1788865853)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
start ost second time...
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg224-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of lustre-ost1/ost1 on ost1 failed 17
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
PASS 1 (114s)

== conf-sanity test 2: start up mds twice (should return err) ========================================================== 07:12:47 (1788865967)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start mds second time..
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 17
Start of lustre-mdt1/mdt1 on mds1 failed 17
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
PASS 2 (105s)

== conf-sanity test 3: mount client twice (should return err) ========================================================== 07:14:32 (1788866072)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
mount.lustre: according to /etc/mtab 192.168.202.124@tcp:/lustre is already mounted on /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
PASS 3 (112s)

== conf-sanity test 4: force cleanup ost, then cleanup === 07:16:24 (1788866184)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
PASS 4 (128s)

== conf-sanity test 5a: force cleanup mds, then cleanup == 07:18:32 (1788866312)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
                     USER        PID ACCESS COMMAND
/mnt/lustre:         root     kernel mount /mnt/lustre
/mnt/lustre is in use by user space process.
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
killing umount
waiting for umount to finish
/home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 149: 27938 Terminated              $UMOUNT -f $MOUNT
manual umount lustre on /mnt/lustre....
umount: /mnt/lustre: not mounted.
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
manual umount lustre on /mnt/lustre....
umount: /mnt/lustre: not mounted.
/etc/mtab updated in 0 secs
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
PASS 5a (140s)

== conf-sanity test 5b: Try to start a client with no MGS (should return errs) ========================================================== 07:20:52 (1788866452)
start ost1 service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
mount.lustre: mount 192.168.202.124@tcp:/lustre at /mnt/lustre failed: Connection timed out
umount lustre on /mnt/lustre.....
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
PASS 5b (289s)

== conf-sanity test 5c: cleanup after failed mount (bug 2712) (should return errs) ========================================================== 07:25:41 (1788866741)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount wrong.lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/wrong.lustre /mnt/lustre
mount.lustre: mount 192.168.202.124@tcp:/wrong.lustre at /mnt/lustre failed: File name too long
umount lustre on /mnt/lustre.....
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
PASS 5c (112s)

== conf-sanity test 5d: mount with ost down ============== 07:27:33 (1788866853)
start ost1 service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
PASS 5d (171s)

== conf-sanity test 5e: delayed connect, don't crash (bug 10268) ========================================================== 07:30:24 (1788867024)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
fail_loc=0x80000506
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
PASS 
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
PASS 5e (125s)

== conf-sanity test 5f: mds down, cleanup after failed mount (bug 2712) ========================================================== 07:32:29 (1788867149)

 SKIP: conf-sanity test_5f needs separate mgs and mds
SKIP 5f (3s)

== conf-sanity test 5g: handle missing debugfs =========== 07:32:32 (1788867152)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
none /sys/kernel/debug debugfs rw,relatime 0 0
PASS 5g (34s)

== conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 07:33:06 (1788867186)
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
Stopping /mnt/lustre-mds1 (opts:) on oleg224-server
fail_loc=0x80000135
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: No such file or directory
oleg224-server: Is the MGS specification correct?
oleg224-server: Is the filesystem name correct?
oleg224-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
Start of lustre-mdt1/mdt1 on mds1 failed 2
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
PASS 5h (125s)

== conf-sanity test 5i: start mdt failure at mdt_quota_init() ========================================================== 07:35:11 (1788867311)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
Stopping /mnt/lustre-mds1 (opts:) on oleg224-server
fail_loc=0x80000A05
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: Bad file descriptor
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 9
Start of lustre-mdt1/mdt1 on mds1 failed 9
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
PASS 5i (153s)

== conf-sanity test 5j: verify libcfs doesn't crash when setting debug flags ========================================================== 07:37:44 (1788867464)
LNET unconfigure error 22: Invalid argument
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
oleg224-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
debug=all
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
PASS 5j (16s)

== conf-sanity test 6: manual umount, then mount again === 07:38:01 (1788867481)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
manual umount lustre on /mnt/lustre....
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
PASS 6 (139s)

== conf-sanity test 7: manual umount, then cleanup ======= 07:40:20 (1788867620)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
manual umount lustre on /mnt/lustre....
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
PASS 7 (130s)

== conf-sanity test 8: double mount setup ================ 07:42:30 (1788867750)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre2.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre2
setup double mount lustre success
umount lustre on /mnt/lustre2.....
Stopping client oleg224-client.virtnet /mnt/lustre2 (opts:)
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
PASS 8 (132s)

== conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================================== 07:44:42 (1788867882)
start ost1 service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
debug=inode trace
subsystem_debug=mds ost
lnet.debug success
lnet.subsystem_debug success
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
PASS 9 (117s)

== conf-sanity test 10a: find lctl param broken symlinks ========================================================== 07:46:39 (1788867999)
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: kmemleak disabled
modules unloaded.
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
PASS 10a (105s)

== conf-sanity test 11: Verify tool --version option works properly ========================================================== 07:48:24 (1788868104)
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
PASS 11 (20s)

== conf-sanity test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 07:48:44 (1788868124)

 SKIP: conf-sanity test_17 ldiskfs only test
SKIP 17 (4s)

== conf-sanity test 18: check mkfs creates large journals ========================================================== 07:48:48 (1788868128)

 SKIP: conf-sanity test_18 ldiskfs only test
SKIP 18 (4s)

== conf-sanity test 19a: start/stop MDS without OSTs ===== 07:48:53 (1788868133)
start mds service on oleg224-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep  8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
PASS 19a (60s)

== conf-sanity test 19b: start/stop OSTs without MDS ===== 07:49:53 (1788868193)
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
PASS 19b (88s)

== conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 07:51:22 (1788868282)
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
setup single mount lustre success
mount lustre with opts remount,ro on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o remount,ro 192.168.202.124@tcp:/lustre /mnt/lustre
touch: cannot touch '/mnt/lustre/f20.conf-sanity': Read-only file system
mount lustre with opts remount,rw on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o remount,rw 192.168.202.124@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
PASS 20 (82s)

== conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 07:52:44 (1788868364)
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
PASS 21a (75s)

== conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 07:53:59 (1788868439)
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
PASS 21b (122s)

== conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 07:56:01 (1788868561)
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost2 service on oleg224-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg224-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop ost2 service on oleg224-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
oleg224-server: 
oleg224-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg224-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 19
oleg224-server: 
oleg224-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg224-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 19
oleg224-server: 
oleg224-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg224-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 19
tunefs.lustre failed, reformatting instead
Stopping clients: oleg224-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg224-client.virtnet /mnt/lustre2 (opts:-f)
oleg224-server: oleg224-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: kmemleak disabled
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1