-----============= acceptance-small: conf-sanity ============----- Sat Sep  5 11:22:02 EDT 2026
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg333-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex
- see CLIENT_VERSION >= v2_15_63-53-g1a6ef725c2 (34683394 >= 34553653) for LU-16938, go 81
- see CLIENT_VERSION >= v2_15_65-31-g74a5b9efaa (34683394 >= 34554143) for LU-18348, go 32b 32c
- see CLIENT_VERSION >= v2_16_55-19-gf69f27cc35 (34683394 >= 34617107) for LU-17920, go 123ae 123ag
- see CLIENT_VERSION >= v2_17_52-94-g86904ef943 (34683394 >= 34681950) for LU-19740, go 115
excepting tests: 102 106 115 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a
skipping tests SLOW=no: 45 69 106 111 114
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg333-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
oleg333-server: oleg333-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg333-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg333-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server

== conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 11:25:05 (1788621905)
start mds service on oleg333-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Change MGS params
max_dirty_mb: 462
new_max_dirty_mb: 452
Waiting 90s for '452'
Updated after 2s: want '452' got '452'
452
Check the value is stored after remount
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:)
Stopping client oleg333-client.virtnet /mnt/lustre opts:
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
Checking servers environments
Checking clients oleg333-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Starting client oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Started clients oleg333-client.virtnet: 
192.168.203.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d0651ce5800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d0651ce5800.idle_timeout=debug
disable quota as required
Change OST params
client_cache_count: 128
new_client_cache_count: 256
Waiting 90s for '256'
Updated after 2s: want '256' got '256'
256
Check the value is stored after remount
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:)
Stopping client oleg333-client.virtnet /mnt/lustre opts:
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
Checking servers environments
Checking clients oleg333-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Starting client oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Started clients oleg333-client.virtnet: 
192.168.203.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d06531a1800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d06531a1800.idle_timeout=debug
disable quota as required
256
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:)
Stopping client oleg333-client.virtnet /mnt/lustre opts:
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
PASS 76a (296s)

== conf-sanity test 76b: verify params log setup correctly ========================================================== 11:30:01 (1788622201)
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:)
Checking servers environments
Checking clients oleg333-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Starting client oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Started clients oleg333-client.virtnet: 
192.168.203.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d0648dd3800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d0648dd3800.idle_timeout=debug
disable quota as required
mgs.MGS.live.params=
fsname: params
flags: 0x20     gen: 1

Secure RPC Config Rules:

imperative_recovery_state:
    state: startup
    nonir_clients: 0
    nidtbl_version: 2
    notify_duration_total: 0.000000000
    notify_duation_max: 0.000000000
    notify_count: 0
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:)
Stopping client oleg333-client.virtnet /mnt/lustre opts:
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
PASS 76b (111s)

== conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 11:31:52 (1788622312)
Checking servers environments
Checking clients oleg333-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Starting client oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Started clients oleg333-client.virtnet: 
192.168.203.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d0648d53800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d0648d53800.idle_timeout=debug
disable quota as required
Change changelog_mask
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Check the value is stored after mds remount
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
start mds service on oleg333-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:)
Stopping client oleg333-client.virtnet /mnt/lustre opts:
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
PASS 76c (147s)

== conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 11:34:19 (1788622459)
Checking servers environments
Checking clients oleg333-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Starting client oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Started clients oleg333-client.virtnet: 
192.168.203.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d0658ba9800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d0658ba9800.idle_timeout=debug
disable quota as required
lctl set_param -P llite.*.xattr_cache=0
Waiting 90s for '0'
Check llite.$inst.xattr_cache on client /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Check llite.$inst.xattr_cache on the new client /mnt/lustre2
mount lustre  on /mnt/lustre2.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre2
umount lustre on /mnt/lustre2.....
Stopping client oleg333-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:)
Stopping client oleg333-client.virtnet /mnt/lustre opts:
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
oleg333-server: No device found for name MGS: Invalid argument
oleg333-server: This command must be run on the MGS.
oleg333-server: error: executing set_param: No such device
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19
Reader error: 'No devices found' at 0
PASS 76d (112s)

== conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 11:36:11 (1788622571)
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1

   Permanent disk data:
Device:     lustre-mdt1_2/mdt1_2
Target:     test1234:MDT0000
Index:      0
Lustre FS:  test1234
Mount type: zfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1_2/mdt1_2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1_2/mdt1_2 properties
  lustre:mgsnode=192.168.203.133@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=101
  lustre:index=0
  lustre:fsname=test1234
  lustre:svname=test1234:MDT0000
Start fs2mds: mount -t lustre -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1_2/mdt1_2
Started test1234-MDT0000

   Permanent disk data:
Device:     lustre-ost1_2/ost1_2
Target:     test1234:OST0000
Index:      0
Lustre FS:  test1234
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=0.0.0.0@tcp,192.168.203.133@tcp,192.168.203.133@tcp:192.168.203.133@tcp:192.168.203.133@tcp sys.timeout=20 failover.node=1.2.3.4@tcp,4.3.2.1@tcp autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1_2/ost1_2
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1_2/ost1_2 properties
  lustre:mgsnode=0.0.0.0@tcp,192.168.203.133@tcp,192.168.203.133@tcp:192.168.203.133@tcp:192.168.203.133@tcp
  lustre:sys.timeout=20
  lustre:failover.node=1.2.3.4@tcp,4.3.2.1@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=test1234
  lustre:svname=test1234:OST0000
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1875: OST_INDICES: bad array subscript
Start fs2ost: mount -t lustre -o localrecov  lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost
seq.cli-test1234-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-ost1_2/ost1_2
Started test1234-OST0000
setup single mount lustre success
umount /mnt/lustre2 ...
stopping fs2mds ...
Stopping /mnt/lustre-fs2mds (opts:-f) on oleg333-server
stopping fs2ost ...
Stopping /mnt/lustre-fs2ost (opts:-f) on oleg333-server
PASS 77 (81s)

== conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 11:37:33 (1788622653)

 SKIP: conf-sanity test_78 ldiskfs only test
SKIP 78 (4s)

== conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 11:37:36 (1788622656)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
oleg333-server: 
oleg333-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg333-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 22
oleg333-server: 
oleg333-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg333-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 22
oleg333-server: 
oleg333-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg333-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 22
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f)
oleg333-server: oleg333-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg333-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg333-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
PASS 79 (186s)

== conf-sanity test 80: mgc import reconnect race ======== 11:40:42 (1788622842)
start mds service on oleg333-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
fail_val=10
fail_loc=0x906
fail_val=10
fail_loc=0x906
start ost2 service on oleg333-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0
fail_loc=0
stop ost2 service on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
PASS 80 (142s)

== conf-sanity test 81: sparse OST indexing ============== 11:43:04 (1788622984)

 SKIP: conf-sanity test_81 needs >= 3 OSTs
SKIP 81 (4s)

== conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 11:43:08 (1788622988)

 SKIP: conf-sanity test_82a needs >= 3 OSTs
SKIP 82a (4s)

== conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 11:43:12 (1788622992)

 SKIP: conf-sanity test_82b needs >= 4 OSTs
SKIP 82b (4s)

== conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 11:43:16 (1788622996)

 SKIP: conf-sanity test_82c needs >= 4 OSTs
SKIP 82c (4s)

== conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 11:43:20 (1788623000)

 SKIP: conf-sanity test_83 ldiskfs only test
SKIP 83 (3s)

== conf-sanity test 84: check recovery_hard_time ========= 11:43:24 (1788623004)
start mds service on oleg333-server
start mds service on oleg333-server
Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg333-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
start ost2 service on oleg333-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0001
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg333-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
recovery_time=60, timeout=20, wrap_up=5
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre2.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre2
UUID                   1K-blocks        Used   Available Use% Mounted on 
lustre-MDT0000_UUID       383616        2048      379520   1% /mnt/lustre[MDT:0] 
lustre-OST0000_UUID       382976        2048      378880   1% /mnt/lustre[OST:0] 
lustre-OST0001_UUID       382976        2048      378880   1% /mnt/lustre[OST:1] 

filesystem_summary:       765952        4096      757760   1% /mnt/lustre 

 - open/close 467 (time 1788623102.77 total 10.01 last 46.65)
 - open/close 955 (time 1788623112.77 total 20.01 last 48.78)
total: 1000 open/close in 21.08 seconds: 47.45 ops/second
fail_loc=0x20000709
fail_val=5
Failing mds1 on oleg333-server
Stopping /mnt/lustre-mds1 (opts:) on oleg333-server
11:45:20 (1788623120) shut down
facet: mds1 facet_host: oleg333-server facet_failover_host: oleg333-server
Failover mds1 to oleg333-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
11:45:53 (1788623153) targets are mounted
11:45:53 (1788623153) facet_failover done
oleg333-client: error: invalid path '/mnt/lustre': Input/output error
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 5
recovery status
status: COMPLETE
recovery_start: 1788623159
recovery_duration: 60
completed_clients: 1/2
replayed_requests: 187
last_transno: 12884902079
VBR: DISABLED
IR: DISABLED
fail_loc=0
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
umount lustre on /mnt/lustre2.....
Stopping client oleg333-client.virtnet /mnt/lustre2 (opts:)
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop ost2 service on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
PASS 84 (254s)

== conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 11:47:38 (1788623258)
fail_loc=0x197
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
PASS 85 (80s)

== conf-sanity test 86: Replacing mkfs.lustre -G option == 11:48:58 (1788623338)

 SKIP: conf-sanity test_86 LU-6442: no such mkfs params for ZFS OSTs
SKIP 86 (3s)

== conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 11:49:01 (1788623341)

 SKIP: conf-sanity test_87 ldiskfs only test
SKIP 87 (3s)

== conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 11:49:04 (1788623344)

 SKIP: conf-sanity test_88a LU-6662: no implementation for ZFS
SKIP 88a (3s)

== conf-sanity test 88b: check the default mount options can be overridden ========================================================== 11:49:07 (1788623347)

 SKIP: conf-sanity test_88b LU-6662: no implementation for ZFS
SKIP 88b (3s)

== conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 11:49:10 (1788623350)
Failing mds1 on oleg333-server
11:49:15 (1788623355) shut down
facet: mds1 facet_host: oleg333-server facet_failover_host: oleg333-server
Failover mds1 to oleg333-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
11:49:38 (1788623378) targets are mounted
11:49:38 (1788623378) facet_failover done
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 95
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 95
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
tunefs --param failover.node=192.0.2.254@tcp0
tunefs --param failover.node=192.0.2.255@tcp0
tunefs --erase-param failover.node
tunefs --erase-params
tunefs --param failover.node=192.0.2.254@tcp0 --erase-params
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f)
oleg333-server: oleg333-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg333-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg333-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
PASS 89 (189s)

== conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 11:52:19 (1788623539)
start mds service on oleg333-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 8 files ...
fail_loc=0x159
launch 6 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-7 has perms 0600 OK
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-8 has perms 0644 OK
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
PASS 90a (117s)

== conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ========================================================== 11:54:16 (1788623656)
start mds service on oleg333-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff9d06529b4800.max_mod_rpcs_in_flight=1
max_mod_rpcs_in_flight set to 1
creating 2 files ...
fail_loc=0x159
launch 0 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-1 has perms 0600 OK
fail_loc=0x159
launch 1 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-2 has perms 0644 OK
mdc.lustre-MDT0000-mdc-ffff9d06529b4800.max_mod_rpcs_in_flight=5
max_mod_rpcs_in_flight set to 5
creating 6 files ...
fail_loc=0x159
launch 4 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-5 has perms 0600 OK
fail_loc=0x159
launch 5 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-6 has perms 0644 OK
mdt_max_mod_rpcs_in_flight is 8
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff9d06458fd800.max_rpcs_in_flight=17
mdc.lustre-MDT0000-mdc-ffff9d06458fd800.max_mod_rpcs_in_flight=16
max_mod_rpcs_in_flight set to 16
creating 17 files ...
fail_loc=0x159
launch 15 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-16 has perms 0600 OK
fail_loc=0x159
launch 16 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-17 has perms 0644 OK
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
PASS 90b (207s)

== conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ========================================================== 11:57:43 (1788623863)
start mds service on oleg333-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
max_rpcs_in_flight is 8
MDC max_mod_rpcs_in_flight is 7
mdt_max_mod_rpcs_in_flight is 8
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mdc.lustre-MDT0000-mdc-ffff9d06531e6800.max_mod_rpcs_in_flight=8
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff9d0648f98000.max_rpcs_in_flight=10
error: set_param: setting /sys/fs/lustre/mdc/lustre-MDT0000-mdc-ffff9d0648f98000/max_mod_rpcs_in_flight=9: Numerical result out of range
error: set_param: setting 'mdc/lustre-MDT0000-mdc-*/max_mod_rpcs_in_flight'='9': Numerical result out of range
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 90c (115s)

== conf-sanity test 90d: check one close RPC is allowed above max_mod_rpcs_in_flight ========================================================== 11:59:38 (1788623978)
start mds service on oleg333-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 7 files ...
multiop /mnt/lustre/d90d.conf-sanity/file-close vO_c
TMPPIPE=/tmp/multiop_open_wait_pipe.7630
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional close in parallel ...
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
PASS 90d (125s)

== conf-sanity test 91: evict-by-nid support ============= 12:01:43 (1788624103)
start mds service on oleg333-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
setup single mount lustre success
list nids on mdt:
mdt.lustre-MDT0000.exports.0@lo
mdt.lustre-MDT0000.exports.192.168.203.33@tcp
mdt.lustre-MDT0000.exports.clear
uuid from 192\.168\.203\.33@tcp:
mdt.lustre-MDT0000.exports.192.168.203.33@tcp.uuid=b5211285-d292-42f0-8bda-0a86861daf93
manual umount lustre on /mnt/lustre....
evict 192\.168\.203\.33@tcp
oleg333-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.203.33@tcp/uuid': No such device
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19
oleg333-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.203.33@tcp/uuid': No such device
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19
oleg333-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.203.33@tcp/uuid': No such device
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19
oleg333-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.203.33@tcp/uuid': No such device
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19
umount lustre on /mnt/lustre.....
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 91 (140s)

== conf-sanity test 92: ldev returns MGS NID correctly in command substitution ========================================================== 12:04:03 (1788624243)
Host is oleg333-client.virtnet
----- /tmp/ldev.conf -----
oleg333-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg333-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg333-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg333-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg333-server	192.168.203.133@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.203.133@tcp
lustre-MGS0000: 192.168.203.133@tcp
lustre-OST0001: 192.168.203.133@tcp
lustre-MDT0000: 192.168.203.133@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.203.133@tcp
lustre-OST0001: 192.168.203.133@tcp
lustre-MDT0000: 192.168.203.133@tcp
lustre-MGS0000: 192.168.203.133@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0001: 192.168.203.133@tcp
lustre-MGS0000: 192.168.203.133@tcp
lustre-MDT0000: 192.168.203.133@tcp
lustre-OST0000: 192.168.203.133@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.203.133@tcp
lustre-MDT0000: 192.168.203.133@tcp
lustre-MGS0000: 192.168.203.133@tcp
lustre-OST0001: 192.168.203.133@tcp
--- END OF LDEV OUTPUT ---
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 92 (9s)

== conf-sanity test 93: register mulitple MDT at the same time ========================================================== 12:04:12 (1788624252)

 SKIP: conf-sanity test_93 needs >= 3 MDTs
SKIP 93 (4s)

== conf-sanity test 94: ldev outputs correct labels for file system name query ========================================================== 12:04:16 (1788624256)
----- /tmp/ldev.conf -----
oleg333-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg333-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg333-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg333-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg333-server	192.168.203.133@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-MDT0000
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 94 (7s)

== conf-sanity test 95: ldev should only allow one label filter ========================================================== 12:04:23 (1788624263)
----- /tmp/ldev.conf -----
oleg333-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg333-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg333-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg333-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg333-server	192.168.203.133@tcp
--- END /tmp/nids ---
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 95 (8s)

== conf-sanity test 96: ldev returns hostname and backend fs correctly in command sub ========================================================== 12:04:31 (1788624271)
----- /tmp/ldev.conf -----
oleg333-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg333-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg333-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg333-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg333-server	192.168.203.133@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
oleg333-server-zfs
oleg333-server-zfs
oleg333-server-zfs
oleg333-server-zfs
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
oleg333-server-zfs
oleg333-server-zfs
oleg333-server-zfs
oleg333-server-zfs
--- END OF EXPECTED OUTPUT ---
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 96 (8s)

== conf-sanity test 97: ldev returns correct ouput when querying based on role ========================================================== 12:04:40 (1788624280)
----- /tmp/ldev.conf -----
oleg333-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg333-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg333-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg333-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg333-server	192.168.203.133@tcp
--- END /tmp/nids ---

MDT role
-- START OF LDEV OUTPUT --
lustre-MDT0000
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
--- END OF EXPECTED OUTPUT ---

OST role
-- START OF LDEV OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---

MGS role
-- START OF LDEV OUTPUT --
lustre-MGS0000
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MGS0000
--- END OF EXPECTED OUTPUT ---
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 97 (10s)

== conf-sanity test 98: Buffer-overflow check while parsing mount_opts ========================================================== 12:04:49 (1788624289)
start mds service on oleg333-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
setup single mount lustre success
arg[0] = mount.lustre
arg[1] = -o
arg[2] = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
arg[3] = -v
arg[4] = 192.168.203.133@tcp:/lustre
arg[5] = /mnt/lustre
source = 192.168.203.133@tcp:/lustre (192.168.203.133@tcp:/lustre), target = /mnt/lustre
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 98 (114s)

== conf-sanity test 99: Adding meta_bg option ============ 12:06:43 (1788624403)

 SKIP: conf-sanity test_99 ldiskfs only test
SKIP 99 (3s)

== conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 12:06:46 (1788624406)
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
oleg333-server: oleg333-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg333-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
setup single mount lustre success
lustre-MDT0000:
lustre-OST0000:
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 100 (150s)

== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:09:16 (1788624556)
start mds service on oleg333-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
 - open/close 228 (time 1788624644.02 total 10.95 last 20.82)
 - open/close 427 (time 1788624654.07 total 21.00 last 19.80)
 - open/close 691 (time 1788624664.58 total 31.51 last 25.12)
 - open/close 938 (time 1788624675.57 total 42.50 last 22.48)
 - open/close 1157 (time 1788624686.02 total 52.95 last 20.96)
 - open/close 1382 (time 1788624696.66 total 63.59 last 21.15)
 - open/close 1562 (time 1788624706.67 total 73.60 last 17.99)
 - open/close 1775 (time 1788624716.67 total 83.60 last 21.29)
 - open/close 2037 (time 1788624726.68 total 93.61 last 26.18)
 - open/close 2199 (time 1788624738.50 total 105.43 last 13.71)
 - open/close 2321 (time 1788624748.51 total 115.44 last 12.18)
 - open/close 2482 (time 1788624758.52 total 125.45 last 16.08)
 - open/close 2701 (time 1788624768.82 total 135.74 last 21.28)
 - open/close 2923 (time 1788624779.53 total 146.46 last 20.72)
 - open/close 3111 (time 1788624789.54 total 156.47 last 18.77)
 - open/close 3334 (time 1788624800.25 total 167.18 last 20.83)
 - open/close 3547 (time 1788624810.93 total 177.86 last 19.94)
 - open/close 3802 (time 1788624821.96 total 188.89 last 23.12)
 - open/close 3989 (time 1788624832.25 total 199.18 last 18.17)
 - open/close 4220 (time 1788624842.26 total 209.19 last 23.08)
 - open/close 4374 (time 1788624852.86 total 219.78 last 14.54)
 - open/close 4501 (time 1788624862.89 total 229.82 last 12.65)
 - open/close 4618 (time 1788624875.00 total 241.93 last 9.66)
 - open/close 4769 (time 1788624885.57 total 252.50 last 14.29)
 - open/close 4832 (time 1788624896.33 total 263.26 last 5.86)
 - open/close 5003 (time 1788624908.35 total 275.28 last 14.22)
 - open/close 5146 (time 1788624919.12 total 286.04 last 13.28)
 - open/close 5245 (time 1788624930.24 total 297.17 last 8.90)
 - open/close 5365 (time 1788624940.81 total 307.74 last 11.36)
 - open/close 5604 (time 1788624950.82 total 317.75 last 23.87)
 - open/close 5840 (time 1788624960.83 total 327.76 last 23.59)
 - open/close 6080 (time 1788624970.83 total 337.76 last 24.00)
 - open/close 6315 (time 1788624981.36 total 348.29 last 22.31)
 - open/close 6574 (time 1788624991.36 total 358.29 last 25.90)
 - open/close 6803 (time 1788625001.77 total 368.70 last 22.00)
 - open/close 7016 (time 1788625011.83 total 378.76 last 21.18)
 - open/close 7255 (time 1788625022.31 total 389.24 last 22.79)
 - open/close 7490 (time 1788625032.35 total 399.28 last 23.41)
 - open/close 7730 (time 1788625042.88 total 409.80 last 22.81)
 - open/close 7899 (time 1788625053.18 total 420.11 last 16.40)
 - open/close 8142 (time 1788625064.20 total 431.13 last 22.05)
 - open/close 8322 (time 1788625074.23 total 441.15 last 17.95)
 - open/close 8522 (time 1788625084.23 total 451.16 last 19.98)
 - open/close 8743 (time 1788625095.19 total 462.12 last 20.17)
 - open/close 8981 (time 1788625105.79 total 472.72 last 22.45)
 - open/close 9221 (time 1788625116.42 total 483.35 last 22.58)
 - open/close 9451 (time 1788625126.58 total 493.51 last 22.64)
 - open/close 9678 (time 1788625137.48 total 504.40 last 20.83)
 - open/close 9874 (time 1788625148.52 total 515.45 last 17.74)
 - open/close 10000 (time 1788625153.82 total 520.75 last 23.77)
 - open/close 10140 (time 1788625163.85 total 530.78 last 13.96)
 - open/close 10239 (time 1788625175.11 total 542.04 last 8.80)
 - open/close 10320 (time 1788625185.34 total 552.26 last 7.92)
 - open/close 10528 (time 1788625197.74 total 564.67 last 16.77)
 - open/close 10597 (time 1788625208.26 total 575.19 last 6.56)
 - open/close 10726 (time 1788625218.99 total 585.92 last 12.02)
 - open/close 10959 (time 1788625229.00 total 595.92 last 23.28)
 - open/close 11102 (time 1788625239.48 total 606.41 last 13.64)
 - open/close 11254 (time 1788625250.25 total 617.18 last 14.12)
 - open/close 11327 (time 1788625260.40 total 627.33 last 7.19)
 - open/close 11364 (time 1788625271.74 total 638.66 last 3.27)
 - open/close 11567 (time 1788625282.30 total 649.23 last 19.22)
 - open/close 11743 (time 1788625292.31 total 659.24 last 17.58)
 - open/close 11907 (time 1788625302.33 total 669.26 last 16.36)
 - open/close 12087 (time 1788625312.34 total 679.26 last 17.99)
 - open/close 12265 (time 1788625322.56 total 689.49 last 17.41)
 - open/close 12479 (time 1788625332.82 total 699.75 last 20.85)
 - open/close 12673 (time 1788625343.00 total 709.93 last 19.07)
 - open/close 12899 (time 1788625353.55 total 720.48 last 21.41)
 - open/close 13161 (time 1788625365.49 total 732.41 last 21.96)
 - open/close 13403 (time 1788625375.86 total 742.79 last 23.32)
 - open/close 13627 (time 1788625385.88 total 752.81 last 22.35)
 - open/close 13839 (time 1788625396.01 total 762.94 last 20.93)
 - open/close 14021 (time 1788625406.02 total 772.95 last 18.18)
 - open/close 14213 (time 1788625418.55 total 785.48 last 15.33)
 - open/close 14402 (time 1788625428.94 total 795.86 last 18.19)
 - open/close 14617 (time 1788625439.23 total 806.15 last 20.89)
 - open/close 14825 (time 1788625449.71 total 816.64 last 19.84)
 - open/close 15101 (time 1788625459.97 total 826.90 last 26.91)
 - open/close 15348 (time 1788625470.93 total 837.86 last 22.54)
 - open/close 15602 (time 1788625481.97 total 848.90 last 22.99)
 - open/close 15816 (time 1788625492.41 total 859.34 last 20.51)
 - open/close 16071 (time 1788625502.41 total 869.34 last 25.49)
 - open/close 16283 (time 1788625513.21 total 880.14 last 19.63)
 - open/close 16358 (time 1788625524.23 total 891.15 last 6.81)
 - open/close 16482 (time 1788625534.26 total 901.19 last 12.35)
 - open/close 16600 (time 1788625544.29 total 911.22 last 11.77)
 - open/close 16821 (time 1788625554.95 total 921.88 last 20.73)
 - open/close 17032 (time 1788625565.32 total 932.25 last 20.35)
 - open/close 17135 (time 1788625576.00 total 942.92 last 9.65)
 - open/close 17313 (time 1788625586.99 total 953.92 last 16.19)
 - open/close 17428 (time 1788625597.57 total 964.49 last 10.87)
 - open/close 17621 (time 1788625607.93 total 974.85 last 18.63)
 - open/close 18092 (time 1788625617.94 total 984.87 last 47.05)
 - open/close 18737 (time 1788625627.94 total 994.87 last 64.50)
 - open/close 19346 (time 1788625637.94 total 1004.87 last 60.87)
 - open/close 20000 (time 1788625647.70 total 1014.63 last 67.02)
 - open/close 20675 (time 1788625657.73 total 1024.66 last 67.32)
 - open/close 21325 (time 1788625667.73 total 1034.66 last 64.96)
 - open/close 21963 (time 1788625677.74 total 1044.67 last 63.78)
 - open/close 22573 (time 1788625687.75 total 1054.68 last 60.90)
 - open/close 23270 (time 1788625697.76 total 1064.69 last 69.67)
 - open/close 23894 (time 1788625707.76 total 1074.69 last 62.37)
 - open/close 24561 (time 1788625717.78 total 1084.71 last 66.59)
 - open/close 25183 (time 1788625727.78 total 1094.71 last 62.18)
 - open/close 25775 (time 1788625737.80 total 1104.72 last 59.13)
 - open/close 26314 (time 1788625747.81 total 1114.74 last 53.83)
 - open/close 26837 (time 1788625757.82 total 1124.74 last 52.26)
 - open/close 27477 (time 1788625767.82 total 1134.75 last 63.97)
 - open/close 28152 (time 1788625777.83 total 1144.76 last 67.43)
 - open/close 28766 (time 1788625787.83 total 1154.76 last 61.38)
 - open/close 29486 (time 1788625797.84 total 1164.77 last 71.94)
 - open/close 30000 (time 1788625805.85 total 1172.78 last 64.18)
 - open/close 30622 (time 1788625815.86 total 1182.78 last 62.17)
 - open/close 31261 (time 1788625825.87 total 1192.80 last 63.81)
 - open/close 31950 (time 1788625835.91 total 1202.83 last 68.66)
 - open/close 32571 (time 1788625845.92 total 1212.84 last 62.04)
 - open/close 33212 (time 1788625855.93 total 1222.85 last 64.04)
 - open/close 33854 (time 1788625865.93 total 1232.86 last 64.15)
 - open/close 34458 (time 1788625875.95 total 1242.87 last 60.34)
 - open/close 35074 (time 1788625885.95 total 1252.88 last 61.56)
 - open/close 35854 (time 1788625895.96 total 1262.89 last 77.91)
 - open/close 36415 (time 1788625905.98 total 1272.91 last 56.00)
 - open/close 36995 (time 1788625915.99 total 1282.92 last 57.96)
 - open/close 37620 (time 1788625925.99 total 1292.92 last 62.46)
 - open/close 38269 (time 1788625936.01 total 1302.93 last 64.82)
 - open/close 38860 (time 1788625946.01 total 1312.94 last 59.06)
 - open/close 39478 (time 1788625956.01 total 1322.94 last 61.80)
 - open/close 40000 (time 1788625964.42 total 1331.35 last 62.06)
 - open/close 40654 (time 1788625974.43 total 1341.36 last 65.34)
 - open/close 41264 (time 1788625984.44 total 1351.36 last 60.98)
 - open/close 41886 (time 1788625994.45 total 1361.38 last 62.09)
 - open/close 42532 (time 1788626004.46 total 1371.39 last 64.55)
 - open/close 43196 (time 1788626014.47 total 1381.40 last 66.31)
 - open/close 43781 (time 1788626024.49 total 1391.42 last 58.42)
 - open/close 44422 (time 1788626034.49 total 1401.42 last 64.05)
 - open/close 45121 (time 1788626044.50 total 1411.43 last 69.87)
 - open/close 45686 (time 1788626054.51 total 1421.43 last 56.46)
 - open/close 46337 (time 1788626064.51 total 1431.44 last 65.05)
 - open/close 46900 (time 1788626074.52 total 1441.45 last 56.26)
 - open/close 47436 (time 1788626084.53 total 1451.45 last 53.57)
 - open/close 48122 (time 1788626094.53 total 1461.46 last 68.56)
 - open/close 48764 (time 1788626104.54 total 1471.47 last 64.11)
 - open/close 49339 (time 1788626114.56 total 1481.49 last 57.40)
 - open/close 49879 (time 1788626124.57 total 1491.49 last 53.98)
total: 50000 open/close in 1493.59 seconds: 33.48 ops/second
 - unlinked 0 (time 1788626130 ; total 0 ; last 0)
 - unlinked 10000 (time 1788626284 ; total 154 ; last 154)
 - unlinked 20000 (time 1788626447 ; total 317 ; last 163)
 - unlinked 30000 (time 1788626607 ; total 477 ; last 160)
 - unlinked 40000 (time 1788626765 ; total 635 ; last 158)
total: 50000 unlinks in 807 seconds: 61.957870 unlinks/second
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
PASS 101a (2424s)

== conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 12:49:41 (1788626981)
start mds service on oleg333-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
fail_loc=0x80002107
fail_val=20
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
oleg333-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d0645f31000.ost_server_uuid 50
osc.lustre-OST0000-osc-ffff9d0645f31000.ost_server_uuid in IDLE state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 101b (197s)

 SKIP: conf-sanity test_102 skipping excluded test 102

== conf-sanity test 103: rename filesystem name ========== 12:52:59 (1788627179)
Checking servers environments
Checking clients oleg333-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Starting client oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Started clients oleg333-client.virtnet: 
192.168.203.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d0653b30000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d0653b30000.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 2s: want 'procname_uid' got 'procname_uid'
disable quota as required
oleg333-server: Pool lustre.pool1 created
oleg333-server: Pool lustre.lustre created
oleg333-server: OST lustre-OST0000_UUID added to pool lustre.lustre
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:)
Stopping client oleg333-client.virtnet /mnt/lustre opts:
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
rename lustre to mylustre
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-mdt1/mdt1
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

   Permanent disk data:
Device:     lustre-mdt1/mdt1
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:svname=mylustre-MDT0000
cmd: zfs set mountpoint=/tmp/mntev5nn7 lustre-mdt1/mdt1 && zfs set canmount=on lustre-mdt1/mdt1 && zfs mount lustre-mdt1/mdt1
cmd: zfs umount lustre-mdt1/mdt1 && zfs set canmount=off lustre-mdt1/mdt1 && zfs set mountpoint=none lustre-mdt1/mdt1
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=mylustre
  lustre:svname=mylustre-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: autodegrade=on mgsnode=192.168.203.133@tcp sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: autodegrade=on mgsnode=192.168.203.133@tcp sys.timeout=20 autodegrade=on
  lustre:svname=mylustre-OST0000
cmd: zfs set mountpoint=/tmp/mntwEwbAh lustre-ost1/ost1 && zfs set canmount=on lustre-ost1/ost1 && zfs mount lustre-ost1/ost1
cmd: zfs umount lustre-ost1/ost1 && zfs set canmount=off lustre-ost1/ost1 && zfs set mountpoint=none lustre-ost1/ost1
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:mgsnode=192.168.203.133@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=mylustre
  lustre:svname=mylustre-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost2/ost2
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.133@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost2/ost2
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.133@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=mylustre-OST0001
cmd: zfs set mountpoint=/tmp/mntMNJg2R lustre-ost2/ost2 && zfs set canmount=on lustre-ost2/ost2 && zfs mount lustre-ost2/ost2
cmd: zfs umount lustre-ost2/ost2 && zfs set canmount=off lustre-ost2/ost2 && zfs set mountpoint=none lustre-ost2/ost2
Writing lustre-ost2/ost2 properties
  lustre:mgsnode=192.168.203.133@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=mylustre
  lustre:svname=mylustre-OST0001
Checking servers environments
Checking clients oleg333-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  mylustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started mylustre-MDT0000
Start ost1: mount -t lustre -o localrecov  mylustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-mylustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started mylustre-OST0000
Start ost2: mount -t lustre -o localrecov  mylustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-mylustre-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started mylustre-OST0001
mount mylustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/mylustre /mnt/lustre
Starting client oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/mylustre /mnt/lustre
Started clients oleg333-client.virtnet: 
192.168.203.133@tcp:/mylustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.mylustre-OST0000-osc-ffff9d0645cdc800.idle_timeout=debug
osc.mylustre-OST0001-osc-ffff9d0645cdc800.idle_timeout=debug
disable quota as required
  File: /mnt/lustre/d103.conf-sanity/test-framework.sh
  Size: 319538    	Blocks: 1027       IO Block: 4194304 regular file
Device: c3aa56ceh/3282720462d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-09-05 12:54:47.000000000 -0400
Modify: 2026-09-05 12:54:48.000000000 -0400
Change: 2026-09-05 12:54:48.000000000 -0400
 Birth: 2026-09-05 12:54:48.000000000 -0400
Pool: mylustre.pool1
Pool: mylustre.lustre
mylustre-OST0000_UUID
mylustre-OST0000_UUID
oleg333-server: OST mylustre-OST0001_UUID added to pool mylustre.lustre
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:)
Stopping client oleg333-client.virtnet /mnt/lustre opts:
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
rename mylustre to tfs
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-mdt1/mdt1
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

   Permanent disk data:
Device:     mylustre-mdt1/mdt1
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:svname=tfs-MDT0000
cmd: zfs set mountpoint=/tmp/mntOb4aqz mylustre-mdt1/mdt1 && zfs set canmount=on mylustre-mdt1/mdt1 && zfs mount mylustre-mdt1/mdt1
cmd: zfs umount mylustre-mdt1/mdt1 && zfs set canmount=off mylustre-mdt1/mdt1 && zfs set mountpoint=none mylustre-mdt1/mdt1
Writing mylustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=tfs
  lustre:svname=tfs-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-ost1/ost1
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: autodegrade=on mgsnode=192.168.203.133@tcp sys.timeout=20

   Permanent disk data:
Device:     mylustre-ost1/ost1
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: autodegrade=on mgsnode=192.168.203.133@tcp sys.timeout=20 autodegrade=on
  lustre:svname=tfs-OST0000
cmd: zfs set mountpoint=/tmp/mntkiCQ7G mylustre-ost1/ost1 && zfs set canmount=on mylustre-ost1/ost1 && zfs mount mylustre-ost1/ost1
cmd: zfs umount mylustre-ost1/ost1 && zfs set canmount=off mylustre-ost1/ost1 && zfs set mountpoint=none mylustre-ost1/ost1
Writing mylustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:mgsnode=192.168.203.133@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=tfs
  lustre:svname=tfs-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-ost2/ost2
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.133@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     mylustre-ost2/ost2
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.133@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=tfs-OST0001
cmd: zfs set mountpoint=/tmp/mntuCXa0C mylustre-ost2/ost2 && zfs set canmount=on mylustre-ost2/ost2 && zfs mount mylustre-ost2/ost2
cmd: zfs umount mylustre-ost2/ost2 && zfs set canmount=off mylustre-ost2/ost2 && zfs set mountpoint=none mylustre-ost2/ost2
Writing mylustre-ost2/ost2 properties
  lustre:mgsnode=192.168.203.133@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=tfs
  lustre:svname=tfs-OST0001
Checking servers environments
Checking clients oleg333-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  tfs-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started tfs-MDT0000
Start ost1: mount -t lustre -o localrecov  tfs-ost1/ost1 /mnt/lustre-ost1
seq.cli-tfs-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started tfs-OST0000
Start ost2: mount -t lustre -o localrecov  tfs-ost2/ost2 /mnt/lustre-ost2
seq.cli-tfs-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started tfs-OST0001
mount tfs  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/tfs /mnt/lustre
Starting client oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/tfs /mnt/lustre
Started clients oleg333-client.virtnet: 
192.168.203.133@tcp:/tfs on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.tfs-OST0000-osc-ffff9d0653b30800.idle_timeout=debug
osc.tfs-OST0001-osc-ffff9d0653b30800.idle_timeout=debug
disable quota as required
  File: /mnt/lustre/d103.conf-sanity/test-framework.sh
  Size: 319538    	Blocks: 1027       IO Block: 4194304 regular file
Device: 32e2fa5ah/853736026d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-09-05 12:54:47.000000000 -0400
Modify: 2026-09-05 12:54:48.000000000 -0400
Change: 2026-09-05 12:54:48.000000000 -0400
 Birth: 2026-09-05 12:54:48.000000000 -0400
Pool: tfs.pool1
Pool: tfs.lustre
tfs-OST0000_UUID
tfs-OST0001_UUID
tfs-OST0000_UUID
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:)
Stopping client oleg333-client.virtnet /mnt/lustre opts:
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
rename tfs to lustre
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-mdt1/mdt1
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

   Permanent disk data:
Device:     tfs-mdt1/mdt1
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:svname=lustre-MDT0000
cmd: zfs set mountpoint=/tmp/mntzBt5rH tfs-mdt1/mdt1 && zfs set canmount=on tfs-mdt1/mdt1 && zfs mount tfs-mdt1/mdt1
cmd: zfs umount tfs-mdt1/mdt1 && zfs set canmount=off tfs-mdt1/mdt1 && zfs set mountpoint=none tfs-mdt1/mdt1
Writing tfs-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-ost1/ost1
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: autodegrade=on mgsnode=192.168.203.133@tcp sys.timeout=20

   Permanent disk data:
Device:     tfs-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: autodegrade=on mgsnode=192.168.203.133@tcp sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
cmd: zfs set mountpoint=/tmp/mntSePhCz tfs-ost1/ost1 && zfs set canmount=on tfs-ost1/ost1 && zfs mount tfs-ost1/ost1
cmd: zfs umount tfs-ost1/ost1 && zfs set canmount=off tfs-ost1/ost1 && zfs set mountpoint=none tfs-ost1/ost1
Writing tfs-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:mgsnode=192.168.203.133@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-ost2/ost2
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.133@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     tfs-ost2/ost2
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.133@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0001
cmd: zfs set mountpoint=/tmp/mntPYmCNJ tfs-ost2/ost2 && zfs set canmount=on tfs-ost2/ost2 && zfs mount tfs-ost2/ost2
cmd: zfs umount tfs-ost2/ost2 && zfs set canmount=off tfs-ost2/ost2 && zfs set mountpoint=none tfs-ost2/ost2
Writing tfs-ost2/ost2 properties
  lustre:mgsnode=192.168.203.133@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=lustre
  lustre:svname=lustre-OST0001
Checking servers environments
Checking clients oleg333-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Starting client oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
Started clients oleg333-client.virtnet: 
192.168.203.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d0648d58000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d0648d58000.idle_timeout=debug
disable quota as required
PASS 103 (474s)

== conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 13:00:53 (1788627653)
mkfs.lustre --mountfsopts: user_xattr,recovery_time_soft=222
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg333-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
oleg333-server: oleg333-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
mount opts: -o localrecov,recovery_time_soft=188
Start mds1: mount -t lustre -o localrecov,recovery_time_soft=188  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
found wanted mdt.lustre-MDT0000.recovery_time_soft=188
PASS 104a (160s)

== conf-sanity test 104b: Mount uses last flock argument ========================================================== 13:03:33 (1788627813)
mount lustre with opts flock,localflock on /mnt/lustre3.....
Starting client: oleg333-client.virtnet:  -o flock,localflock 192.168.203.133@tcp:/lustre /mnt/lustre3
192.168.203.133@tcp:/lustre on /mnt/lustre3 type lustre (rw,localflock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,nouser_xattr,verbose)
umount lustre on /mnt/lustre3.....
Stopping client oleg333-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock on /mnt/lustre3.....
Starting client: oleg333-client.virtnet:  -o localflock,flock 192.168.203.133@tcp:/lustre /mnt/lustre3
192.168.203.133@tcp:/lustre on /mnt/lustre3 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,nouser_xattr,verbose)
umount lustre on /mnt/lustre3.....
Stopping client oleg333-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock,noflock on /mnt/lustre3.....
Starting client: oleg333-client.virtnet:  -o localflock,flock,noflock 192.168.203.133@tcp:/lustre /mnt/lustre3
umount lustre on /mnt/lustre3.....
Stopping client oleg333-client.virtnet /mnt/lustre3 (opts:)
PASS 104b (15s)

== conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 13:03:48 (1788627828)
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: lustre_rmmod: wait 40s for 5 OBD devices
oleg333-server: lustre_rmmod: OBD devices after 60s:
oleg333-server:   2 UP mgc MGC192.168.203.133@tcp 6ffbc8af-bb84-4f54-a072-5466b942f2ad 5
oleg333-server:  10 UP ost OSS OSS_uuid 3
oleg333-server:  14 UP osd-zfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5
oleg333-server:  15 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 3
oleg333-server: kmemleak disabled
oleg333-server:  16 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5
oleg333-server: rmmod: ERROR: Module mdd is in use
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server
oleg333-server: oleg333-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg333-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg333-client.virtnet:  -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre
touch: cannot touch '/tmp/d105.conf-sanity/f105.conf-sanity': Read-only file system
umount lustre on /mnt/lustre.....
Stopping client oleg333-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 105 (270s)

 SKIP: conf-sanity test_106 skipping SLOW test 106

== conf-sanity test 107: Unknown config param should not fail target mounting ========================================================== 13:08:20 (1788628100)
start mds service on oleg333-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
start ost1 service on oleg333-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1
Started lustre-OST0000
oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
umount lustre on /mnt/lustre.....
stop ost1 service on oleg333-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server
stop mds service on oleg333-server
unloading modules via unload_modules_local on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing unload_modules_local
oleg333-server: kmemleak disabled
modules unloaded.
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
PASS 107 (216s)

 SKIP: conf-sanity test_108a skipping ALWAYS excluded test 108a

== conf-sanity test 108b: migrate from ZFS to ldiskfs ==== 13:11:59 (1788628319)

 SKIP: conf-sanity test_108b ldiskfs only test
SKIP 108b (4s)

== conf-sanity test 109a: test lctl clear_conf fsname ==== 13:12:03 (1788628323)

 SKIP: conf-sanity test_109a LU-8727: no implementation for ZFS
SKIP 109a (4s)

== conf-sanity test 109b: test lctl clear_conf one config ========================================================== 13:12:07 (1788628327)

 SKIP: conf-sanity test_109b LU-8727: no implementation for ZFS
SKIP 109b (4s)

== conf-sanity test 110: Adding large_dir with 3-level htree ========================================================== 13:12:11 (1788628331)

 SKIP: conf-sanity test_110 Only applicable to ldiskfs-based MDTs
SKIP 110 (3s)

 SKIP: conf-sanity test_111 skipping SLOW test 111

 SKIP: conf-sanity test_112a skipping ALWAYS excluded test 112a

 SKIP: conf-sanity test_112b skipping ALWAYS excluded test 112b

 SKIP: conf-sanity test_113 skipping ALWAYS excluded test 113

 SKIP: conf-sanity test_114 skipping SLOW test 114

 SKIP: conf-sanity test_115 skipping excluded test 115

== conf-sanity test 116: big size MDT support ============ 13:12:25 (1788628345)

 SKIP: conf-sanity test_116 ldiskfs only test
SKIP 116 (3s)

 SKIP: conf-sanity test_117 skipping ALWAYS excluded test 117

 SKIP: conf-sanity test_119 skipping ALWAYS excluded test 119

== conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 13:12:32 (1788628352)

 SKIP: conf-sanity test_120 mdt count < 2
SKIP 120 (3s)
Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2
oleg333-server: oleg333-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg333-server'
oleg333-server: oleg333-server.virtnet: executing load_modules_local
oleg333-server: kmemleak disabled
oleg333-server: Loading modules from /home/green/git/lustre-release/lustre
oleg333-server: detected 4 online CPUs by sysfs
oleg333-server: MODOPTS_LIBCFS=
oleg333-server: Force libcfs to create 2 CPU partitions
oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
== conf-sanity test complete, duration 6702 sec ========== 13:13:45 (1788628425)
=== conf-sanity: start cleanup 13:13:47 (1788628427) ===
=== conf-sanity: finish cleanup 13:13:51 (1788628431) ===