-----============= acceptance-small: conf-sanity ============----- Mon Jun  1 13:02:29 EDT 2026
mgs: Rocky Linux release 8.10 (Green Obsidian)
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
mds1: Rocky Linux release 8.10 (Green Obsidian)
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
ost1: Rocky Linux release 8.10 (Green Obsidian)
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
client: Rocky Linux release 8.10 (Green Obsidian)
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg225-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory
excepting tests: 102 106 115 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a
skipping tests SLOW=no: 45 69 106 111 114
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg225-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
oleg225-server: oleg225-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

== conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 13:05:46 (1780333546)
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Change MGS params
max_dirty_mb: 462
new_max_dirty_mb: 452
Waiting 90s for '452'
Updated after 3s: want '452' got '452'
452
Check the value is stored after remount
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:)
Stopping client oleg225-client.virtnet /mnt/lustre opts:
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Checking servers environments
Checking clients oleg225-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Starting client oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Started clients oleg225-client.virtnet: 
192.168.202.125@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9660465e9000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9660465e9000.idle_timeout=debug
disable quota as required
Change OST params
client_cache_count: 128
new_client_cache_count: 256
Waiting 90s for '256'
Updated after 2s: want '256' got '256'
256
Check the value is stored after remount
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:)
Stopping client oleg225-client.virtnet /mnt/lustre opts:
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
Checking servers environments
Checking clients oleg225-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Starting client oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Started clients oleg225-client.virtnet: 
192.168.202.125@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff966053826000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff966053826000.idle_timeout=debug
disable quota as required
256
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:)
Stopping client oleg225-client.virtnet /mnt/lustre opts:
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
PASS 76a (323s)

== conf-sanity test 76b: verify params log setup correctly ========================================================== 13:11:09 (1780333869)
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:)
Checking servers environments
Checking clients oleg225-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Starting client oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Started clients oleg225-client.virtnet: 
192.168.202.125@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff966048548800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff966048548800.idle_timeout=debug
disable quota as required
mgs.MGS.live.params=
fsname: params
flags: 0x20     gen: 1

Secure RPC Config Rules:

imperative_recovery_state:
    state: startup
    nonir_clients: 0
    nidtbl_version: 2
    notify_duration_total: 0.000000000
    notify_duation_max: 0.000000000
    notify_count: 0
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:)
Stopping client oleg225-client.virtnet /mnt/lustre opts:
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
PASS 76b (118s)

== conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 13:13:07 (1780333987)
Checking servers environments
Checking clients oleg225-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Starting client oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Started clients oleg225-client.virtnet: 
192.168.202.125@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9660465ee000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9660465ee000.idle_timeout=debug
disable quota as required
Change changelog_mask
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Check the value is stored after mds remount
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:)
Stopping client oleg225-client.virtnet /mnt/lustre opts:
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
PASS 76c (143s)

== conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 13:15:30 (1780334130)
Checking servers environments
Checking clients oleg225-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Starting client oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Started clients oleg225-client.virtnet: 
192.168.202.125@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9660463c6000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9660463c6000.idle_timeout=debug
disable quota as required
lctl set_param -P llite.*.xattr_cache=0
Waiting 90s for '0'
Check llite.$inst.xattr_cache on client /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Check llite.$inst.xattr_cache on the new client /mnt/lustre2
mount lustre  on /mnt/lustre2.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre2
umount lustre on /mnt/lustre2.....
Stopping client oleg225-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:)
Stopping client oleg225-client.virtnet /mnt/lustre opts:
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
oleg225-server: No device found for name MGS: Invalid argument
oleg225-server: This command must be run on the MGS.
oleg225-server: error: executing set_param: No such device
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 19
Reader error: 'No devices found' at 0
PASS 76d (115s)

== conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 13:17:25 (1780334245)
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1

   Permanent disk data:
Device:     lustre-mdt1_2/mdt1_2
Target:     test1234:MDT0000
Index:      0
Lustre FS:  test1234
Mount type: zfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1_2/mdt1_2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1_2/mdt1_2 properties
  lustre:mgsnode=192.168.202.125@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=101
  lustre:index=0
  lustre:fsname=test1234
  lustre:svname=test1234:MDT0000
Start fs2mds: mount -t lustre -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1_2/mdt1_2
Started test1234-MDT0000

   Permanent disk data:
Device:     lustre-ost1_2/ost1_2
Target:     test1234:OST0000
Index:      0
Lustre FS:  test1234
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=0.0.0.0@tcp,192.168.202.125@tcp,192.168.202.125@tcp:192.168.202.125@tcp:192.168.202.125@tcp sys.timeout=20 failover.node=1.2.3.4@tcp,4.3.2.1@tcp autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1_2/ost1_2
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1_2/ost1_2 properties
  lustre:mgsnode=0.0.0.0@tcp,192.168.202.125@tcp,192.168.202.125@tcp:192.168.202.125@tcp:192.168.202.125@tcp
  lustre:sys.timeout=20
  lustre:failover.node=1.2.3.4@tcp,4.3.2.1@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=test1234
  lustre:svname=test1234:OST0000
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1839: OST_INDICES: bad array subscript
Start fs2ost: mount -t lustre -o localrecov  lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost
seq.cli-test1234-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-ost1_2/ost1_2
Started test1234-OST0000
setup single mount lustre success
umount /mnt/lustre2 ...
stopping fs2mds ...
Stopping /mnt/lustre-fs2mds (opts:-f) on oleg225-server
stopping fs2ost ...
Stopping /mnt/lustre-fs2ost (opts:-f) on oleg225-server
PASS 77 (84s)

== conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 13:18:49 (1780334329)

 SKIP: conf-sanity test_78 ldiskfs only test
SKIP 78 (3s)

== conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 13:18:52 (1780334332)
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: 
oleg225-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg225-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 22
oleg225-server: 
oleg225-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg225-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 22
oleg225-server: 
oleg225-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg225-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 22
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f)
oleg225-server: oleg225-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
PASS 79 (180s)

== conf-sanity test 80: mgc import reconnect race ======== 13:21:52 (1780334512)
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
fail_val=10
fail_loc=0x906
fail_val=10
fail_loc=0x906
start ost2 service on oleg225-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
fail_loc=0
stop ost2 service on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
PASS 80 (141s)

== conf-sanity test 81: sparse OST indexing ============== 13:24:13 (1780334653)

 SKIP: conf-sanity test_81 needs >= 3 OSTs
SKIP 81 (4s)

== conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 13:24:17 (1780334657)

 SKIP: conf-sanity test_82a needs >= 3 OSTs
SKIP 82a (6s)

== conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 13:24:23 (1780334663)

 SKIP: conf-sanity test_82b needs >= 4 OSTs
SKIP 82b (6s)

== conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 13:24:29 (1780334669)

 SKIP: conf-sanity test_82c needs >= 4 OSTs
SKIP 82c (5s)

== conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 13:24:34 (1780334674)

 SKIP: conf-sanity test_83 ldiskfs only test
SKIP 83 (5s)

== conf-sanity test 84: check recovery_hard_time ========= 13:24:39 (1780334679)
start mds service on oleg225-server
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
start ost2 service on oleg225-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0001
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg225-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
recovery_time=60, timeout=20, wrap_up=5
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre2.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre2
UUID                   1K-blocks        Used   Available Use%  Mounted on
lustre-MDT0000_UUID       383616        2048      379520   1%  /mnt/lustre[MDT:0] 
lustre-OST0000_UUID       382976        2048      378880   1%  /mnt/lustre[OST:0] 
lustre-OST0001_UUID       382976        2048      378880   1%  /mnt/lustre[OST:1] 

filesystem_summary:       765952        4096      757760   1%  /mnt/lustre 

 - open/close 407 (time 1780334788.06 total 10.01 last 40.67)
 - open/close 793 (time 1780334798.07 total 20.02 last 38.54)
total: 1000 open/close in 24.63 seconds: 40.60 ops/second
fail_loc=0x20000709
fail_val=5
Failing mds1 on oleg225-server
Stopping /mnt/lustre-mds1 (opts:) on oleg225-server
13:26:51 (1780334811) shut down
facet: mds1 facet_host: oleg225-server facet_failover_host: oleg225-server
Failover mds1 to oleg225-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
13:27:16 (1780334836) targets are mounted
13:27:16 (1780334836) facet_failover done
oleg225-client: error: invalid path '/mnt/lustre': Input/output error
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 5
recovery status
status: COMPLETE
recovery_start: 1780334835
recovery_duration: 60
completed_clients: 1/2
replayed_requests: 183
last_transno: 12884902075
VBR: DISABLED
IR: DISABLED
fail_loc=0
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
umount lustre on /mnt/lustre2.....
Stopping client oleg225-client.virtnet /mnt/lustre2 (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop ost2 service on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
PASS 84 (270s)

== conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 13:29:09 (1780334949)
fail_loc=0x197
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
PASS 85 (84s)

== conf-sanity test 86: Replacing mkfs.lustre -G option == 13:30:33 (1780335033)

 SKIP: conf-sanity test_86 LU-6442: no such mkfs params for ZFS OSTs
SKIP 86 (4s)

== conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 13:30:37 (1780335037)

 SKIP: conf-sanity test_87 ldiskfs only test
SKIP 87 (4s)

== conf-sanity test 88: check the default mount options can be overridden ========================================================== 13:30:41 (1780335041)

 SKIP: conf-sanity test_88 LU-6662: no implementation for ZFS
SKIP 88 (6s)

== conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 13:30:48 (1780335048)
Failing mds1 on oleg225-server
13:30:56 (1780335056) shut down
facet: mds1 facet_host: oleg225-server facet_failover_host: oleg225-server
Failover mds1 to oleg225-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
13:31:21 (1780335081) targets are mounted
13:31:21 (1780335081) facet_failover done
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 95
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 95
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
tunefs --param failover.node=192.0.2.254@tcp0
tunefs --param failover.node=192.0.2.255@tcp0
tunefs --erase-param failover.node
tunefs --erase-params
tunefs --param failover.node=192.0.2.254@tcp0 --erase-params
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f)
oleg225-server: oleg225-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
PASS 89 (215s)

== conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 13:34:22 (1780335262)
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 8 files ...
fail_loc=0x159
launch 6 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-7 has perms 0600 OK
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-8 has perms 0644 OK
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 90a (121s)

== conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ========================================================== 13:36:23 (1780335383)
start mds service on oleg225-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff9660463c7000.max_mod_rpcs_in_flight=1
max_mod_rpcs_in_flight set to 1
creating 2 files ...
fail_loc=0x159
launch 0 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-1 has perms 0600 OK
fail_loc=0x159
launch 1 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-2 has perms 0644 OK
mdc.lustre-MDT0000-mdc-ffff9660463c7000.max_mod_rpcs_in_flight=5
max_mod_rpcs_in_flight set to 5
creating 6 files ...
fail_loc=0x159
launch 4 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-5 has perms 0600 OK
fail_loc=0x159
launch 5 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-6 has perms 0644 OK
mdt_max_mod_rpcs_in_flight is 8
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff966048160000.max_rpcs_in_flight=17
mdc.lustre-MDT0000-mdc-ffff966048160000.max_mod_rpcs_in_flight=16
max_mod_rpcs_in_flight set to 16
creating 17 files ...
fail_loc=0x159
launch 15 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-16 has perms 0600 OK
fail_loc=0x159
launch 16 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-17 has perms 0644 OK
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
PASS 90b (231s)

== conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ========================================================== 13:40:14 (1780335614)
start mds service on oleg225-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
max_rpcs_in_flight is 8
MDC max_mod_rpcs_in_flight is 7
mdt_max_mod_rpcs_in_flight is 8
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mdc.lustre-MDT0000-mdc-ffff966053826800.max_mod_rpcs_in_flight=8
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff96604a261000.max_rpcs_in_flight=10
error: set_param: setting /sys/fs/lustre/mdc/lustre-MDT0000-mdc-ffff96604a261000/max_mod_rpcs_in_flight=9: Numerical result out of range
error: set_param: setting 'mdc/lustre-MDT0000-mdc-*/max_mod_rpcs_in_flight'='9': Numerical result out of range
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
PASS 90c (119s)

== conf-sanity test 90d: check one close RPC is allowed above max_mod_rpcs_in_flight ========================================================== 13:42:13 (1780335733)
start mds service on oleg225-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 7 files ...
multiop /mnt/lustre/d90d.conf-sanity/file-close vO_c
TMPPIPE=/tmp/multiop_open_wait_pipe.7575
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional close in parallel ...
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 90d (140s)

== conf-sanity test 91: evict-by-nid support ============= 13:44:33 (1780335873)
start mds service on oleg225-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
setup single mount lustre success
list nids on mdt:
mdt.lustre-MDT0000.exports.0@lo
mdt.lustre-MDT0000.exports.192.168.202.25@tcp
mdt.lustre-MDT0000.exports.clear
uuid from 192\.168\.202\.25@tcp:
mdt.lustre-MDT0000.exports.192.168.202.25@tcp.uuid=e2b98b18-eb4a-4c18-9a2b-c7b8deb05200
manual umount lustre on /mnt/lustre....
evict 192\.168\.202\.25@tcp
oleg225-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.202.25@tcp/uuid': No such device
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 19
oleg225-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.202.25@tcp/uuid': No such device
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 19
oleg225-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.202.25@tcp/uuid': No such device
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 19
oleg225-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.202.25@tcp/uuid': No such device
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 19
umount lustre on /mnt/lustre.....
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 91 (164s)

== conf-sanity test 92: ldev returns MGS NID correctly in command substitution ========================================================== 13:47:17 (1780336037)
Host is oleg225-client.virtnet
----- /tmp/ldev.conf -----
oleg225-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg225-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg225-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg225-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg225-server	192.168.202.125@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-MGS0000: 192.168.202.125@tcp
lustre-MDT0000: 192.168.202.125@tcp
lustre-OST0001: 192.168.202.125@tcp
lustre-OST0000: 192.168.202.125@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0001: 192.168.202.125@tcp
lustre-MGS0000: 192.168.202.125@tcp
lustre-OST0000: 192.168.202.125@tcp
lustre-MDT0000: 192.168.202.125@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-MGS0000: 192.168.202.125@tcp
lustre-OST0001: 192.168.202.125@tcp
lustre-OST0000: 192.168.202.125@tcp
lustre-MDT0000: 192.168.202.125@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.202.125@tcp
lustre-MDT0000: 192.168.202.125@tcp
lustre-OST0001: 192.168.202.125@tcp
lustre-MGS0000: 192.168.202.125@tcp
--- END OF LDEV OUTPUT ---
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 92 (10s)

== conf-sanity test 93: register mulitple MDT at the same time ========================================================== 13:47:27 (1780336047)

 SKIP: conf-sanity test_93 needs >= 3 MDTs
SKIP 93 (4s)

== conf-sanity test 94: ldev outputs correct labels for file system name query ========================================================== 13:47:31 (1780336051)
----- /tmp/ldev.conf -----
oleg225-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg225-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg225-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg225-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg225-server	192.168.202.125@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-MDT0000
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 94 (8s)

== conf-sanity test 95: ldev should only allow one label filter ========================================================== 13:47:39 (1780336059)
----- /tmp/ldev.conf -----
oleg225-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg225-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg225-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg225-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg225-server	192.168.202.125@tcp
--- END /tmp/nids ---
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 95 (10s)

== conf-sanity test 96: ldev returns hostname and backend fs correctly in command sub ========================================================== 13:47:49 (1780336069)
----- /tmp/ldev.conf -----
oleg225-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg225-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg225-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg225-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg225-server	192.168.202.125@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
oleg225-server-zfs
oleg225-server-zfs
oleg225-server-zfs
oleg225-server-zfs
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
oleg225-server-zfs
oleg225-server-zfs
oleg225-server-zfs
oleg225-server-zfs
--- END OF EXPECTED OUTPUT ---
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 96 (10s)

== conf-sanity test 97: ldev returns correct ouput when querying based on role ========================================================== 13:47:59 (1780336079)
----- /tmp/ldev.conf -----
oleg225-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg225-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg225-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg225-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg225-server	192.168.202.125@tcp
--- END /tmp/nids ---

MDT role
-- START OF LDEV OUTPUT --
lustre-MDT0000
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
--- END OF EXPECTED OUTPUT ---

OST role
-- START OF LDEV OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---

MGS role
-- START OF LDEV OUTPUT --
lustre-MGS0000
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MGS0000
--- END OF EXPECTED OUTPUT ---
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 97 (10s)

== conf-sanity test 98: Buffer-overflow check while parsing mount_opts ========================================================== 13:48:09 (1780336089)
start mds service on oleg225-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
setup single mount lustre success
arg[0] = mount.lustre
arg[1] = -o
arg[2] = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
arg[3] = -v
arg[4] = 192.168.202.125@tcp:/lustre
arg[5] = /mnt/lustre
source = 192.168.202.125@tcp:/lustre (192.168.202.125@tcp:/lustre), target = /mnt/lustre
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
PASS 98 (110s)

== conf-sanity test 99: Adding meta_bg option ============ 13:49:59 (1780336199)

 SKIP: conf-sanity test_99 ldiskfs only test
SKIP 99 (4s)

== conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 13:50:03 (1780336203)
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
oleg225-server: oleg225-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
setup single mount lustre success
lustre-MDT0000:
lustre-OST0000:
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
PASS 100 (170s)

== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:52:53 (1780336373)
start mds service on oleg225-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
 - open/close 161 (time 1780336464.44 total 10.03 last 16.06)
 - open/close 322 (time 1780336474.46 total 20.05 last 16.06)
 - open/close 429 (time 1780336484.49 total 30.08 last 10.67)
 - open/close 447 (time 1780336495.47 total 41.06 last 1.64)
 - open/close 641 (time 1780336505.47 total 51.07 last 19.39)
 - open/close 829 (time 1780336515.78 total 61.37 last 18.25)
 - open/close 1003 (time 1780336525.95 total 71.54 last 17.11)
 - open/close 1195 (time 1780336536.96 total 82.56 last 17.43)
 - open/close 1394 (time 1780336546.97 total 92.56 last 19.89)
 - open/close 1487 (time 1780336558.47 total 104.06 last 8.09)
 - open/close 1646 (time 1780336568.58 total 114.17 last 15.73)
 - open/close 1882 (time 1780336579.66 total 125.25 last 21.31)
 - open/close 2043 (time 1780336589.95 total 135.54 last 15.64)
 - open/close 2279 (time 1780336600.01 total 145.60 last 23.47)
 - open/close 2474 (time 1780336610.29 total 155.88 last 18.96)
 - open/close 2531 (time 1780336621.33 total 166.92 last 5.17)
 - open/close 2736 (time 1780336631.34 total 176.94 last 20.46)
 - open/close 2953 (time 1780336642.13 total 187.72 last 20.12)
 - open/close 3132 (time 1780336653.11 total 198.70 last 16.31)
 - open/close 3227 (time 1780336664.18 total 209.77 last 8.58)
 - open/close 3281 (time 1780336675.00 total 220.59 last 4.99)
 - open/close 3420 (time 1780336685.80 total 231.39 last 12.88)
 - open/close 3654 (time 1780336696.86 total 242.45 last 21.16)
 - open/close 3890 (time 1780336707.19 total 252.78 last 22.84)
 - open/close 4143 (time 1780336717.43 total 263.02 last 24.72)
 - open/close 4298 (time 1780336728.18 total 273.77 last 14.41)
 - open/close 4466 (time 1780336738.99 total 284.58 last 15.54)
 - open/close 4612 (time 1780336749.56 total 295.15 last 13.82)
 - open/close 4753 (time 1780336760.05 total 305.64 last 13.44)
 - open/close 4813 (time 1780336770.42 total 316.01 last 5.78)
 - open/close 4915 (time 1780336781.27 total 326.86 last 9.40)
 - open/close 5037 (time 1780336791.87 total 337.46 last 11.52)
 - open/close 5172 (time 1780336801.90 total 347.49 last 13.45)
 - open/close 5283 (time 1780336812.88 total 358.47 last 10.11)
 - open/close 5378 (time 1780336822.95 total 368.54 last 9.43)
 - open/close 5501 (time 1780336834.81 total 380.40 last 10.38)
 - open/close 5701 (time 1780336845.17 total 390.76 last 19.30)
 - open/close 5859 (time 1780336855.59 total 401.18 last 15.16)
 - open/close 6074 (time 1780336865.59 total 411.18 last 21.49)
 - open/close 6238 (time 1780336877.25 total 422.84 last 14.07)
 - open/close 6464 (time 1780336887.25 total 432.85 last 22.60)
 - open/close 6700 (time 1780336897.27 total 442.86 last 23.56)
 - open/close 6851 (time 1780336908.00 total 453.59 last 14.08)
 - open/close 6950 (time 1780336918.01 total 463.60 last 9.89)
 - open/close 7093 (time 1780336928.65 total 474.24 last 13.44)
 - open/close 7212 (time 1780336939.46 total 485.05 last 11.01)
 - open/close 7417 (time 1780336950.38 total 495.97 last 18.76)
 - open/close 7651 (time 1780336961.32 total 506.91 last 21.40)
 - open/close 7900 (time 1780336971.32 total 516.91 last 24.89)
 - open/close 8055 (time 1780336981.32 total 526.91 last 15.50)
 - open/close 8182 (time 1780336991.35 total 536.94 last 12.67)
 - open/close 8412 (time 1780337001.84 total 547.43 last 21.92)
 - open/close 8579 (time 1780337012.41 total 558.00 last 15.80)
 - open/close 8799 (time 1780337023.36 total 568.95 last 20.09)
 - open/close 9007 (time 1780337034.07 total 579.66 last 19.43)
 - open/close 9234 (time 1780337044.09 total 589.68 last 22.66)
 - open/close 9406 (time 1780337054.09 total 599.68 last 17.19)
 - open/close 9617 (time 1780337065.14 total 610.73 last 19.11)
 - open/close 9806 (time 1780337075.95 total 621.54 last 17.48)
 - open/close 9903 (time 1780337087.45 total 633.04 last 8.44)
 - open/close 10000 (time 1780337093.95 total 639.54 last 14.92)
 - open/close 10136 (time 1780337105.01 total 650.60 last 12.29)
 - open/close 10309 (time 1780337115.03 total 660.62 last 17.26)
 - open/close 10419 (time 1780337125.04 total 670.63 last 10.99)
 - open/close 10641 (time 1780337135.75 total 681.34 last 20.73)
 - open/close 10876 (time 1780337145.77 total 691.36 last 23.46)
 - open/close 11005 (time 1780337155.92 total 701.51 last 12.70)
 - open/close 11123 (time 1780337165.93 total 711.52 last 11.79)
 - open/close 11279 (time 1780337175.94 total 721.53 last 15.59)
 - open/close 11454 (time 1780337185.96 total 731.55 last 17.46)
 - open/close 11705 (time 1780337196.74 total 742.33 last 23.28)
 - open/close 11986 (time 1780337206.76 total 752.35 last 28.06)
 - open/close 12192 (time 1780337219.14 total 764.73 last 16.63)
 - open/close 12463 (time 1780337230.02 total 775.61 last 24.91)
 - open/close 12692 (time 1780337240.92 total 786.51 last 21.01)
 - open/close 12850 (time 1780337250.92 total 796.51 last 15.80)
 - open/close 13037 (time 1780337262.22 total 807.81 last 16.55)
 - open/close 13267 (time 1780337272.23 total 817.82 last 22.97)
 - open/close 13439 (time 1780337282.75 total 828.34 last 16.36)
 - open/close 13614 (time 1780337292.76 total 838.35 last 17.47)
 - open/close 13876 (time 1780337302.77 total 848.36 last 26.18)
 - open/close 14158 (time 1780337312.82 total 858.41 last 28.07)
 - open/close 14379 (time 1780337322.82 total 868.41 last 22.10)
 - open/close 14446 (time 1780337333.10 total 878.69 last 6.52)
 - open/close 14626 (time 1780337344.12 total 889.71 last 16.33)
 - open/close 14827 (time 1780337354.44 total 900.03 last 19.48)
 - open/close 15008 (time 1780337364.45 total 910.04 last 18.08)
 - open/close 15279 (time 1780337375.48 total 921.07 last 24.57)
 - open/close 15504 (time 1780337385.96 total 931.55 last 21.48)
 - open/close 15712 (time 1780337395.96 total 941.55 last 20.79)
 - open/close 15920 (time 1780337405.97 total 951.56 last 20.78)
 - open/close 16045 (time 1780337416.11 total 961.70 last 12.34)
 - open/close 16264 (time 1780337426.12 total 971.72 last 21.86)
 - open/close 16491 (time 1780337436.14 total 981.73 last 22.66)
 - open/close 16670 (time 1780337446.27 total 991.86 last 17.67)
 - open/close 16832 (time 1780337456.31 total 1001.90 last 16.14)
 - open/close 17004 (time 1780337466.35 total 1011.94 last 17.14)
 - open/close 17182 (time 1780337476.49 total 1022.08 last 17.55)
 - open/close 17458 (time 1780337486.49 total 1032.08 last 27.59)
 - open/close 17944 (time 1780337496.50 total 1042.09 last 48.56)
 - open/close 18478 (time 1780337506.52 total 1052.11 last 53.32)
 - open/close 19055 (time 1780337516.54 total 1062.13 last 57.58)
 - open/close 19704 (time 1780337526.55 total 1072.14 last 64.84)
 - open/close 20000 (time 1780337530.24 total 1075.83 last 80.15)
 - open/close 20765 (time 1780337540.24 total 1085.83 last 76.47)
 - open/close 21478 (time 1780337550.25 total 1095.84 last 71.22)
 - open/close 22154 (time 1780337560.27 total 1105.86 last 67.48)
 - open/close 22820 (time 1780337570.28 total 1115.87 last 66.57)
 - open/close 23551 (time 1780337580.30 total 1125.89 last 72.96)
 - open/close 24229 (time 1780337590.30 total 1135.89 last 67.80)
 - open/close 25038 (time 1780337600.31 total 1145.90 last 80.82)
 - open/close 25804 (time 1780337610.32 total 1155.91 last 76.52)
 - open/close 26642 (time 1780337620.33 total 1165.92 last 83.69)
 - open/close 27381 (time 1780337630.34 total 1175.93 last 73.81)
 - open/close 28222 (time 1780337640.36 total 1185.95 last 83.94)
 - open/close 28905 (time 1780337650.37 total 1195.96 last 68.26)
 - open/close 29406 (time 1780337660.39 total 1205.98 last 49.98)
 - open/close 29919 (time 1780337670.39 total 1215.98 last 51.29)
 - open/close 30546 (time 1780337680.40 total 1225.99 last 62.64)
 - open/close 31176 (time 1780337690.41 total 1236.01 last 62.93)
 - open/close 31909 (time 1780337700.42 total 1246.01 last 73.29)
 - open/close 32672 (time 1780337710.42 total 1256.01 last 76.28)
 - open/close 33425 (time 1780337720.43 total 1266.02 last 75.20)
 - open/close 34071 (time 1780337730.43 total 1276.02 last 64.60)
 - open/close 34753 (time 1780337740.43 total 1286.02 last 68.19)
 - open/close 35503 (time 1780337750.44 total 1296.03 last 74.94)
 - open/close 36147 (time 1780337760.45 total 1306.04 last 64.35)
 - open/close 36783 (time 1780337770.47 total 1316.06 last 63.50)
 - open/close 37496 (time 1780337780.47 total 1326.06 last 71.28)
 - open/close 38217 (time 1780337790.47 total 1336.06 last 72.07)
 - open/close 38908 (time 1780337800.48 total 1346.07 last 69.03)
 - open/close 39601 (time 1780337810.49 total 1356.09 last 69.20)
 - open/close 40000 (time 1780337815.63 total 1361.22 last 77.71)
 - open/close 40631 (time 1780337825.63 total 1371.22 last 63.07)
 - open/close 41252 (time 1780337835.63 total 1381.22 last 62.10)
 - open/close 41955 (time 1780337845.63 total 1391.23 last 70.29)
 - open/close 42553 (time 1780337855.65 total 1401.24 last 59.73)
 - open/close 43242 (time 1780337865.66 total 1411.25 last 68.79)
 - open/close 43915 (time 1780337875.68 total 1421.27 last 67.19)
 - open/close 44625 (time 1780337885.71 total 1431.30 last 70.78)
 - open/close 45133 (time 1780337895.72 total 1441.31 last 50.75)
 - open/close 45589 (time 1780337905.75 total 1451.34 last 45.45)
 - open/close 46173 (time 1780337915.76 total 1461.35 last 58.36)
 - open/close 46817 (time 1780337925.77 total 1471.36 last 64.36)
 - open/close 47472 (time 1780337935.77 total 1481.36 last 65.44)
 - open/close 48126 (time 1780337945.78 total 1491.37 last 65.36)
 - open/close 48857 (time 1780337955.80 total 1501.39 last 72.95)
 - open/close 49525 (time 1780337965.80 total 1511.39 last 66.79)
total: 50000 open/close in 1519.84 seconds: 32.90 ops/second
 - unlinked 0 (time 1780337980 ; total 0 ; last 0)
 - unlinked 10000 (time 1780338147 ; total 167 ; last 166)
 - unlinked 20000 (time 1780338308 ; total 328 ; last 161)
 - unlinked 30000 (time 1780338462 ; total 482 ; last 154)
 - unlinked 40000 (time 1780338616 ; total 636 ; last 154)
total: 50000 unlinks in 789 seconds: 63.371357 unlinks/second
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 101a (2435s)

== conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 14:33:28 (1780338808)
start mds service on oleg225-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
fail_loc=0x80002107
fail_val=20
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
oleg225-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9660469d2000.ost_server_uuid 50
osc.lustre-OST0000-osc-ffff9660469d2000.ost_server_uuid in FULL state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 101b (170s)

 SKIP: conf-sanity test_102 skipping excluded test 102

== conf-sanity test 103: rename filesystem name ========== 14:36:20 (1780338980)
Checking servers environments
Checking clients oleg225-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Starting client oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Started clients oleg225-client.virtnet: 
192.168.202.125@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff966045768800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff966045768800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
disable quota as required
oleg225-server: Pool lustre.pool1 created
oleg225-server: Pool lustre.lustre created
oleg225-server: OST lustre-OST0000_UUID added to pool lustre.lustre
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:)
Stopping client oleg225-client.virtnet /mnt/lustre opts:
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
rename lustre to mylustre
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-mdt1/mdt1
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20

   Permanent disk data:
Device:     lustre-mdt1/mdt1
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20
  lustre:svname=mylustre-MDT0000
cmd: zfs set mountpoint=/tmp/mntdktd75 lustre-mdt1/mdt1 && zfs set canmount=on lustre-mdt1/mdt1 && zfs mount lustre-mdt1/mdt1
cmd: zfs umount lustre-mdt1/mdt1 && zfs set canmount=off lustre-mdt1/mdt1 && zfs set mountpoint=none lustre-mdt1/mdt1
Writing lustre-mdt1/mdt1 properties
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:sys.timeout=20
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=mylustre
  lustre:svname=mylustre-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.125@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.125@tcp autodegrade=on
  lustre:svname=mylustre-OST0000
cmd: zfs set mountpoint=/tmp/mntSFQDc7 lustre-ost1/ost1 && zfs set canmount=on lustre-ost1/ost1 && zfs mount lustre-ost1/ost1
cmd: zfs umount lustre-ost1/ost1 && zfs set canmount=off lustre-ost1/ost1 && zfs set mountpoint=none lustre-ost1/ost1
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.125@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=mylustre
  lustre:svname=mylustre-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost2/ost2
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost2/ost2
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=mylustre-OST0001
cmd: zfs set mountpoint=/tmp/mntZ9XQ69 lustre-ost2/ost2 && zfs set canmount=on lustre-ost2/ost2 && zfs mount lustre-ost2/ost2
cmd: zfs umount lustre-ost2/ost2 && zfs set canmount=off lustre-ost2/ost2 && zfs set mountpoint=none lustre-ost2/ost2
Writing lustre-ost2/ost2 properties
  lustre:mgsnode=192.168.202.125@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=mylustre
  lustre:svname=mylustre-OST0001
Checking servers environments
Checking clients oleg225-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  mylustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started mylustre-MDT0000
Start ost1: mount -t lustre -o localrecov  mylustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-mylustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started mylustre-OST0000
Start ost2: mount -t lustre -o localrecov  mylustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-mylustre-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started mylustre-OST0001
mount mylustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/mylustre /mnt/lustre
Starting client oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/mylustre /mnt/lustre
Started clients oleg225-client.virtnet: 
192.168.202.125@tcp:/mylustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.mylustre-OST0000-osc-ffff9660463c0000.idle_timeout=debug
osc.mylustre-OST0001-osc-ffff9660463c0000.idle_timeout=debug
disable quota as required
  File: /mnt/lustre/d103.conf-sanity/test-framework.sh
  Size: 313989    	Blocks: 1027       IO Block: 4194304 regular file
Device: c3aa56ceh/3282720462d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-06-01 14:38:02.000000000 -0400
Modify: 2026-06-01 14:38:02.000000000 -0400
Change: 2026-06-01 14:38:02.000000000 -0400
 Birth: 2026-06-01 14:38:02.000000000 -0400
Pool: mylustre.pool1
Pool: mylustre.lustre
mylustre-OST0000_UUID
mylustre-OST0000_UUID
oleg225-server: OST mylustre-OST0001_UUID added to pool mylustre.lustre
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:)
Stopping client oleg225-client.virtnet /mnt/lustre opts:
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
rename mylustre to tfs
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-mdt1/mdt1
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20

   Permanent disk data:
Device:     mylustre-mdt1/mdt1
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20
  lustre:svname=tfs-MDT0000
cmd: zfs set mountpoint=/tmp/mnthh5Y7t mylustre-mdt1/mdt1 && zfs set canmount=on mylustre-mdt1/mdt1 && zfs mount mylustre-mdt1/mdt1
cmd: zfs umount mylustre-mdt1/mdt1 && zfs set canmount=off mylustre-mdt1/mdt1 && zfs set mountpoint=none mylustre-mdt1/mdt1
Writing mylustre-mdt1/mdt1 properties
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:sys.timeout=20
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=tfs
  lustre:svname=tfs-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-ost1/ost1
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.125@tcp

   Permanent disk data:
Device:     mylustre-ost1/ost1
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.125@tcp autodegrade=on
  lustre:svname=tfs-OST0000
cmd: zfs set mountpoint=/tmp/mntQldo2r mylustre-ost1/ost1 && zfs set canmount=on mylustre-ost1/ost1 && zfs mount mylustre-ost1/ost1
cmd: zfs umount mylustre-ost1/ost1 && zfs set canmount=off mylustre-ost1/ost1 && zfs set mountpoint=none mylustre-ost1/ost1
Writing mylustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.125@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=tfs
  lustre:svname=tfs-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-ost2/ost2
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     mylustre-ost2/ost2
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=tfs-OST0001
cmd: zfs set mountpoint=/tmp/mntracGm4 mylustre-ost2/ost2 && zfs set canmount=on mylustre-ost2/ost2 && zfs mount mylustre-ost2/ost2
cmd: zfs umount mylustre-ost2/ost2 && zfs set canmount=off mylustre-ost2/ost2 && zfs set mountpoint=none mylustre-ost2/ost2
Writing mylustre-ost2/ost2 properties
  lustre:mgsnode=192.168.202.125@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=tfs
  lustre:svname=tfs-OST0001
Checking servers environments
Checking clients oleg225-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  tfs-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started tfs-MDT0000
Start ost1: mount -t lustre -o localrecov  tfs-ost1/ost1 /mnt/lustre-ost1
seq.cli-tfs-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started tfs-OST0000
Start ost2: mount -t lustre -o localrecov  tfs-ost2/ost2 /mnt/lustre-ost2
seq.cli-tfs-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started tfs-OST0001
mount tfs  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/tfs /mnt/lustre
Starting client oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/tfs /mnt/lustre
Started clients oleg225-client.virtnet: 
192.168.202.125@tcp:/tfs on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.tfs-OST0000-osc-ffff966042fd7000.idle_timeout=debug
osc.tfs-OST0001-osc-ffff966042fd7000.idle_timeout=debug
disable quota as required
  File: /mnt/lustre/d103.conf-sanity/test-framework.sh
  Size: 313989    	Blocks: 1027       IO Block: 4194304 regular file
Device: 32e2fa5ah/853736026d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-06-01 14:38:02.000000000 -0400
Modify: 2026-06-01 14:38:02.000000000 -0400
Change: 2026-06-01 14:38:02.000000000 -0400
 Birth: 2026-06-01 14:38:02.000000000 -0400
Pool: tfs.pool1
Pool: tfs.lustre
tfs-OST0000_UUID
tfs-OST0001_UUID
tfs-OST0000_UUID
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:)
Stopping client oleg225-client.virtnet /mnt/lustre opts:
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
rename tfs to lustre
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-mdt1/mdt1
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20

   Permanent disk data:
Device:     tfs-mdt1/mdt1
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20
  lustre:svname=lustre-MDT0000
cmd: zfs set mountpoint=/tmp/mntVIGeZJ tfs-mdt1/mdt1 && zfs set canmount=on tfs-mdt1/mdt1 && zfs mount tfs-mdt1/mdt1
cmd: zfs umount tfs-mdt1/mdt1 && zfs set canmount=off tfs-mdt1/mdt1 && zfs set mountpoint=none tfs-mdt1/mdt1
Writing tfs-mdt1/mdt1 properties
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:sys.timeout=20
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-ost1/ost1
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.125@tcp

   Permanent disk data:
Device:     tfs-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.125@tcp autodegrade=on
  lustre:svname=lustre-OST0000
cmd: zfs set mountpoint=/tmp/mntUQqCIb tfs-ost1/ost1 && zfs set canmount=on tfs-ost1/ost1 && zfs mount tfs-ost1/ost1
cmd: zfs umount tfs-ost1/ost1 && zfs set canmount=off tfs-ost1/ost1 && zfs set mountpoint=none tfs-ost1/ost1
Writing tfs-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.125@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-ost2/ost2
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     tfs-ost2/ost2
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.125@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0001
cmd: zfs set mountpoint=/tmp/mntU4Voj5 tfs-ost2/ost2 && zfs set canmount=on tfs-ost2/ost2 && zfs mount tfs-ost2/ost2
cmd: zfs umount tfs-ost2/ost2 && zfs set canmount=off tfs-ost2/ost2 && zfs set mountpoint=none tfs-ost2/ost2
Writing tfs-ost2/ost2 properties
  lustre:mgsnode=192.168.202.125@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=lustre
  lustre:svname=lustre-OST0001
Checking servers environments
Checking clients oleg225-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Starting client oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
Started clients oleg225-client.virtnet: 
192.168.202.125@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff966046593800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff966046593800.idle_timeout=debug
disable quota as required
PASS 103 (453s)

== conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 14:43:53 (1780339433)
mountfsopt: acl,user_xattr
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg225-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
oleg225-server: oleg225-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
Start mds1: mount -t lustre -o localrecov,noacl  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
setfacl: /mnt/lustre: Operation not supported
PASS 104a (182s)

== conf-sanity test 104b: Mount uses last flock argument ========================================================== 14:46:56 (1780339616)
mount lustre with opts flock,localflock on /mnt/lustre3.....
Starting client: oleg225-client.virtnet:  -o flock,localflock 192.168.202.125@tcp:/lustre /mnt/lustre3
192.168.202.125@tcp:/lustre on /mnt/lustre3 type lustre (rw,localflock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose)
umount lustre on /mnt/lustre3.....
Stopping client oleg225-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock on /mnt/lustre3.....
Starting client: oleg225-client.virtnet:  -o localflock,flock 192.168.202.125@tcp:/lustre /mnt/lustre3
192.168.202.125@tcp:/lustre on /mnt/lustre3 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose)
umount lustre on /mnt/lustre3.....
Stopping client oleg225-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock,noflock on /mnt/lustre3.....
Starting client: oleg225-client.virtnet:  -o localflock,flock,noflock 192.168.202.125@tcp:/lustre /mnt/lustre3
umount lustre on /mnt/lustre3.....
Stopping client oleg225-client.virtnet /mnt/lustre3 (opts:)
PASS 104b (16s)

== conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 14:47:13 (1780339633)
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
oleg225-server: rmmod: ERROR: Module mdd is in use
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
oleg225-server: oleg225-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
touch: cannot touch '/tmp/d105.conf-sanity/f105.conf-sanity': Read-only file system
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 105 (196s)

 SKIP: conf-sanity test_106 skipping SLOW test 106

== conf-sanity test 107: Unknown config param should not fail target mounting ========================================================== 14:50:31 (1780339831)
start mds service on oleg225-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
PASS 107 (244s)

 SKIP: conf-sanity test_108a skipping ALWAYS excluded test 108a

== conf-sanity test 108b: migrate from ZFS to ldiskfs ==== 14:54:37 (1780340077)

 SKIP: conf-sanity test_108b ldiskfs only test
SKIP 108b (4s)

== conf-sanity test 109a: test lctl clear_conf fsname ==== 14:54:42 (1780340082)

 SKIP: conf-sanity test_109a LU-8727: no implementation for ZFS
SKIP 109a (4s)

== conf-sanity test 109b: test lctl clear_conf one config ========================================================== 14:54:46 (1780340086)

 SKIP: conf-sanity test_109b LU-8727: no implementation for ZFS
SKIP 109b (4s)

== conf-sanity test 110: Adding large_dir with 3-level htree ========================================================== 14:54:50 (1780340090)

 SKIP: conf-sanity test_110 Only applicable to ldiskfs-based MDTs
SKIP 110 (4s)

 SKIP: conf-sanity test_111 skipping SLOW test 111

 SKIP: conf-sanity test_112a skipping ALWAYS excluded test 112a

 SKIP: conf-sanity test_112b skipping ALWAYS excluded test 112b

 SKIP: conf-sanity test_113 skipping ALWAYS excluded test 113

 SKIP: conf-sanity test_114 skipping SLOW test 114

 SKIP: conf-sanity test_115 skipping excluded test 115

== conf-sanity test 116: big size MDT support ============ 14:55:07 (1780340107)

 SKIP: conf-sanity test_116 ldiskfs only test
SKIP 116 (4s)

 SKIP: conf-sanity test_117 skipping ALWAYS excluded test 117

 SKIP: conf-sanity test_119 skipping ALWAYS excluded test 119

== conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 14:55:15 (1780340115)

 SKIP: conf-sanity test_120 mdt count < 2
SKIP 120 (4s)
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2
oleg225-server: oleg225-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
== conf-sanity test complete, duration 6853 sec ========== 14:56:44 (1780340204)
=== conf-sanity: start cleanup 14:56:46 (1780340206) ===
=== conf-sanity: finish cleanup 14:56:50 (1780340210) ===