-----============= acceptance-small: conf-sanity ============----- Mon Aug 24 17:11:28 EDT 2026
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg439-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex
- see CLIENT_VERSION >= v2_15_63-53-g1a6ef725c2 (34683216 >= 34553653) for LU-16938, go 81
- see CLIENT_VERSION >= v2_15_65-31-g74a5b9efaa (34683216 >= 34554143) for LU-18348, go 32b 32c
- see CLIENT_VERSION >= v2_16_55-19-gf69f27cc35 (34683216 >= 34617107) for LU-17920, go 123ae 123ag
excepting tests: 102 106 115 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a
skipping tests SLOW=no: 45 69 106 111 114
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg439-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
oleg439-server: oleg439-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg439-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg439-server: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg439-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg439-server: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg439-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

== conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 17:15:56 (1787606156)
start mds service on oleg439-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Change MGS params
max_dirty_mb: 462
new_max_dirty_mb: 452
Waiting 90s for '452'
Updated after 2s: want '452' got '452'
452
Check the value is stored after remount
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:)
Stopping client oleg439-client.virtnet /mnt/lustre opts:
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
Checking servers environments
Checking clients oleg439-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Starting client oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Started clients oleg439-client.virtnet: 
192.168.204.139@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff97f2057fb000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff97f2057fb000.idle_timeout=debug
disable quota as required
Change OST params
client_cache_count: 128
new_client_cache_count: 256
Waiting 90s for '256'
Updated after 3s: want '256' got '256'
256
Check the value is stored after remount
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:)
Stopping client oleg439-client.virtnet /mnt/lustre opts:
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
Checking servers environments
Checking clients oleg439-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Starting client oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Started clients oleg439-client.virtnet: 
192.168.204.139@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff97f205535000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff97f205535000.idle_timeout=debug
disable quota as required
256
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:)
Stopping client oleg439-client.virtnet /mnt/lustre opts:
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
PASS 76a (452s)

== conf-sanity test 76b: verify params log setup correctly ========================================================== 17:23:28 (1787606608)
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:)
Checking servers environments
Checking clients oleg439-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Starting client oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Started clients oleg439-client.virtnet: 
192.168.204.139@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff97f205433000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff97f205433000.idle_timeout=debug
disable quota as required
mgs.MGS.live.params=
fsname: params
flags: 0x20     gen: 1

Secure RPC Config Rules:

imperative_recovery_state:
    state: startup
    nonir_clients: 0
    nidtbl_version: 2
    notify_duration_total: 0.000000000
    notify_duation_max: 0.000000000
    notify_count: 0
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:)
Stopping client oleg439-client.virtnet /mnt/lustre opts:
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
PASS 76b (154s)

== conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 17:26:02 (1787606762)
Checking servers environments
Checking clients oleg439-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Starting client oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Started clients oleg439-client.virtnet: 
192.168.204.139@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff97f207202000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff97f207202000.idle_timeout=debug
disable quota as required
Change changelog_mask
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Check the value is stored after mds remount
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
start mds service on oleg439-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:)
Stopping client oleg439-client.virtnet /mnt/lustre opts:
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
PASS 76c (229s)

== conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 17:29:52 (1787606992)
Checking servers environments
Checking clients oleg439-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Starting client oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Started clients oleg439-client.virtnet: 
192.168.204.139@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff97f205530800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff97f205530800.idle_timeout=debug
disable quota as required
lctl set_param -P llite.*.xattr_cache=0
Waiting 90s for '0'
Check llite.$inst.xattr_cache on client /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Check llite.$inst.xattr_cache on the new client /mnt/lustre2
mount lustre  on /mnt/lustre2.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2
umount lustre on /mnt/lustre2.....
Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:)
Stopping client oleg439-client.virtnet /mnt/lustre opts:
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
oleg439-server: No device found for name MGS: Invalid argument
oleg439-server: This command must be run on the MGS.
oleg439-server: error: executing set_param: No such device
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 19
Reader error: 'No devices found' at 0
PASS 76d (152s)

== conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 17:32:23 (1787607143)
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1

   Permanent disk data:
Device:     lustre-mdt1_2/mdt1_2
Target:     test1234:MDT0000
Index:      0
Lustre FS:  test1234
Mount type: zfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.139@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1_2/mdt1_2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1_2/mdt1_2 properties
  lustre:mgsnode=192.168.204.139@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=101
  lustre:index=0
  lustre:fsname=test1234
  lustre:svname=test1234:MDT0000
Start fs2mds: mount -t lustre -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1_2/mdt1_2
Started test1234-MDT0000

   Permanent disk data:
Device:     lustre-ost1_2/ost1_2
Target:     test1234:OST0000
Index:      0
Lustre FS:  test1234
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=0.0.0.0@tcp,192.168.204.139@tcp,192.168.204.139@tcp:192.168.204.139@tcp:192.168.204.139@tcp sys.timeout=20 failover.node=1.2.3.4@tcp,4.3.2.1@tcp autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1_2/ost1_2
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1_2/ost1_2 properties
  lustre:mgsnode=0.0.0.0@tcp,192.168.204.139@tcp,192.168.204.139@tcp:192.168.204.139@tcp:192.168.204.139@tcp
  lustre:sys.timeout=20
  lustre:failover.node=1.2.3.4@tcp,4.3.2.1@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=test1234
  lustre:svname=test1234:OST0000
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1850: OST_INDICES: bad array subscript
Start fs2ost: mount -t lustre -o localrecov  lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost
seq.cli-test1234-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-ost1_2/ost1_2
Started test1234-OST0000
setup single mount lustre success
umount /mnt/lustre2 ...
stopping fs2mds ...
Stopping /mnt/lustre-fs2mds (opts:-f) on oleg439-server
stopping fs2ost ...
Stopping /mnt/lustre-fs2ost (opts:-f) on oleg439-server
PASS 77 (89s)

== conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 17:33:52 (1787607232)

 SKIP: conf-sanity test_78 ldiskfs only test
SKIP 78 (4s)

== conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 17:33:56 (1787607236)
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: 
oleg439-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg439-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22
oleg439-server: 
oleg439-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg439-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22
oleg439-server: 
oleg439-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg439-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:-f)
oleg439-server: oleg439-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg439-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg439-server: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg439-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg439-server: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg439-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
PASS 79 (260s)

== conf-sanity test 80: mgc import reconnect race ======== 17:38:16 (1787607496)
start mds service on oleg439-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
fail_val=10
fail_loc=0x906
fail_val=10
fail_loc=0x906
start ost2 service on oleg439-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0
fail_loc=0
stop ost2 service on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
PASS 80 (170s)

== conf-sanity test 81: sparse OST indexing ============== 17:41:06 (1787607666)

 SKIP: conf-sanity test_81 needs >= 3 OSTs
SKIP 81 (4s)

== conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 17:41:10 (1787607670)

 SKIP: conf-sanity test_82a needs >= 3 OSTs
SKIP 82a (3s)

== conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 17:41:13 (1787607673)

 SKIP: conf-sanity test_82b needs >= 4 OSTs
SKIP 82b (3s)

== conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 17:41:16 (1787607676)

 SKIP: conf-sanity test_82c needs >= 4 OSTs
SKIP 82c (3s)

== conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 17:41:19 (1787607679)

 SKIP: conf-sanity test_83 ldiskfs only test
SKIP 83 (4s)

== conf-sanity test 84: check recovery_hard_time ========= 17:41:23 (1787607683)
start mds service on oleg439-server
start mds service on oleg439-server
Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg439-server: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg439-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg439-server: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg439-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
start ost2 service on oleg439-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0001
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg439-server: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg439-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg439-server: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg439-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
recovery_time=60, timeout=20, wrap_up=5
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre2.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2
UUID                   1K-blocks        Used   Available Use% Mounted on 
lustre-MDT0000_UUID       383616        2048      379520   1% /mnt/lustre[MDT:0] 
lustre-MDT0001_UUID       383616        1920      379648   1% /mnt/lustre[MDT:1] 
lustre-OST0000_UUID       382976        2048      378880   1% /mnt/lustre[OST:0] 
lustre-OST0001_UUID       382976        2048      378880   1% /mnt/lustre[OST:1] 

filesystem_summary:       765952        4096      757760   1% /mnt/lustre 

 - open/close 463 (time 1787607817.90 total 10.03 last 46.17)
 - open/close 919 (time 1787607827.91 total 20.03 last 45.59)
total: 1000 open/close in 21.60 seconds: 46.29 ops/second
fail_loc=0x20000709
fail_val=5
Failing mds1 on oleg439-server
Stopping /mnt/lustre-mds1 (opts:) on oleg439-server
17:43:57 (1787607837) shut down
facet: mds1 facet_host: oleg439-server facet_failover_host: oleg439-server
Failover mds1 to oleg439-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
17:44:23 (1787607863) targets are mounted
17:44:23 (1787607863) facet_failover done
oleg439-client: error: invalid path '/mnt/lustre': Input/output error
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 5
recovery status
status: COMPLETE
recovery_start: 1787607859
recovery_duration: 60
completed_clients: 2/3
replayed_requests: 185
last_transno: 12884902077
VBR: DISABLED
IR: DISABLED
fail_loc=0
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
umount lustre on /mnt/lustre2.....
Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:)
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop ost2 service on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
PASS 84 (288s)

== conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 17:46:11 (1787607971)
fail_loc=0x197
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
PASS 85 (81s)

== conf-sanity test 86: Replacing mkfs.lustre -G option == 17:47:32 (1787608052)

 SKIP: conf-sanity test_86 LU-6442: no such mkfs params for ZFS OSTs
SKIP 86 (4s)

== conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 17:47:36 (1787608056)

 SKIP: conf-sanity test_87 ldiskfs only test
SKIP 87 (4s)

== conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 17:47:41 (1787608061)

 SKIP: conf-sanity test_88a LU-6662: no implementation for ZFS
SKIP 88a (4s)

== conf-sanity test 88b: check the default mount options can be overridden ========================================================== 17:47:45 (1787608065)

 SKIP: conf-sanity test_88b LU-6662: no implementation for ZFS
SKIP 88b (4s)

== conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 17:47:50 (1787608070)
Failing mds1 on oleg439-server
17:47:55 (1787608075) shut down
facet: mds1 facet_host: oleg439-server facet_failover_host: oleg439-server
Failover mds1 to oleg439-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
17:48:20 (1787608100) targets are mounted
17:48:20 (1787608100) facet_failover done
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 95
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 95
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
tunefs --param failover.node=192.0.2.254@tcp0
tunefs --param failover.node=192.0.2.255@tcp0
tunefs --erase-param failover.node
tunefs --erase-params
tunefs --param failover.node=192.0.2.254@tcp0 --erase-params
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:-f)
oleg439-server: oleg439-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg439-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg439-server: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg439-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg439-server: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg439-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
PASS 89 (288s)

== conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 17:52:37 (1787608357)
start mds service on oleg439-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 8 files ...
fail_loc=0x159
launch 6 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-7 has perms 0600 OK
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-8 has perms 0644 OK
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
PASS 90a (167s)

== conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ========================================================== 17:55:24 (1787608524)
start mds service on oleg439-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff97f208289800.max_mod_rpcs_in_flight=1
max_mod_rpcs_in_flight set to 1
creating 2 files ...
fail_loc=0x159
launch 0 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-1 has perms 0600 OK
fail_loc=0x159
launch 1 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-2 has perms 0644 OK
mdc.lustre-MDT0001-mdc-ffff97f208289800.max_mod_rpcs_in_flight=5
max_mod_rpcs_in_flight set to 5
creating 6 files ...
fail_loc=0x159
launch 4 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-5 has perms 0600 OK
fail_loc=0x159
launch 5 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-6 has perms 0644 OK
mdt_max_mod_rpcs_in_flight is 8
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff97f204cc3800.max_rpcs_in_flight=17
mdc.lustre-MDT0000-mdc-ffff97f204cc3800.max_mod_rpcs_in_flight=16
max_mod_rpcs_in_flight set to 16
creating 17 files ...
fail_loc=0x159
launch 15 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-16 has perms 0600 OK
fail_loc=0x159
launch 16 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-17 has perms 0644 OK
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
PASS 90b (281s)

== conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ========================================================== 18:00:05 (1787608805)
start mds service on oleg439-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
max_rpcs_in_flight is 8
MDC max_mod_rpcs_in_flight is 7
mdt_max_mod_rpcs_in_flight is 8
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mdc.lustre-MDT0000-mdc-ffff97f2186c9000.max_mod_rpcs_in_flight=8
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff97f20528b800.max_rpcs_in_flight=10
error: set_param: setting /sys/fs/lustre/mdc/lustre-MDT0000-mdc-ffff97f20528b800/max_mod_rpcs_in_flight=9: Numerical result out of range
error: set_param: setting 'mdc/lustre-MDT0000-mdc-*/max_mod_rpcs_in_flight'='9': Numerical result out of range
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
PASS 90c (172s)

== conf-sanity test 90d: check one close RPC is allowed above max_mod_rpcs_in_flight ========================================================== 18:02:57 (1787608977)
start mds service on oleg439-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 7 files ...
multiop /mnt/lustre/d90d.conf-sanity/file-close vO_c
TMPPIPE=/tmp/multiop_open_wait_pipe.8436
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional close in parallel ...
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
PASS 90d (183s)

== conf-sanity test 91: evict-by-nid support ============= 18:06:00 (1787609160)
start mds service on oleg439-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
setup single mount lustre success
list nids on mdt:
mdt.lustre-MDT0000.exports.0@lo
mdt.lustre-MDT0000.exports.192.168.204.39@tcp
mdt.lustre-MDT0000.exports.clear
mdt.lustre-MDT0001.exports.0@lo
mdt.lustre-MDT0001.exports.192.168.204.39@tcp
mdt.lustre-MDT0001.exports.clear
uuid from 192\.168\.204\.39@tcp:
mdt.lustre-MDT0000.exports.192.168.204.39@tcp.uuid=00d9956a-686d-4d25-b568-65a625bc6de7
mdt.lustre-MDT0001.exports.192.168.204.39@tcp.uuid=00d9956a-686d-4d25-b568-65a625bc6de7
manual umount lustre on /mnt/lustre....
evict 192\.168\.204\.39@tcp
oleg439-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.204.39@tcp/uuid': No such device
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 19
oleg439-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.204.39@tcp/uuid': No such device
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 19
oleg439-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.204.39@tcp/uuid': No such device
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 19
oleg439-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.204.39@tcp/uuid': No such device
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 19
umount lustre on /mnt/lustre.....
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
PASS 91 (194s)

== conf-sanity test 92: ldev returns MGS NID correctly in command substitution ========================================================== 18:09:15 (1787609355)
Host is oleg439-client.virtnet
----- /tmp/ldev.conf -----
oleg439-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg439-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg439-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg439-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
oleg439-server	-	lustre-MDT0001	zfs:lustre-mdt2/mdt2
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg439-server	192.168.204.139@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-MDT0000: 192.168.204.139@tcp
lustre-OST0000: 192.168.204.139@tcp
lustre-MDT0001: 192.168.204.139@tcp
lustre-MGS0000: 192.168.204.139@tcp
lustre-OST0001: 192.168.204.139@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.204.139@tcp
lustre-MDT0000: 192.168.204.139@tcp
lustre-OST0001: 192.168.204.139@tcp
lustre-MGS0000: 192.168.204.139@tcp
lustre-MDT0001: 192.168.204.139@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0001: 192.168.204.139@tcp
lustre-OST0000: 192.168.204.139@tcp
lustre-MDT0001: 192.168.204.139@tcp
lustre-MGS0000: 192.168.204.139@tcp
lustre-MDT0000: 192.168.204.139@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.204.139@tcp
lustre-MGS0000: 192.168.204.139@tcp
lustre-MDT0001: 192.168.204.139@tcp
lustre-OST0001: 192.168.204.139@tcp
lustre-MDT0000: 192.168.204.139@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-MGS0000: 192.168.204.139@tcp
lustre-OST0000: 192.168.204.139@tcp
lustre-MDT0000: 192.168.204.139@tcp
lustre-OST0001: 192.168.204.139@tcp
lustre-MDT0001: 192.168.204.139@tcp
--- END OF LDEV OUTPUT ---
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
PASS 92 (10s)

== conf-sanity test 93: register mulitple MDT at the same time ========================================================== 18:09:25 (1787609365)

 SKIP: conf-sanity test_93 needs >= 3 MDTs
SKIP 93 (4s)

== conf-sanity test 94: ldev outputs correct labels for file system name query ========================================================== 18:09:29 (1787609369)
----- /tmp/ldev.conf -----
oleg439-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg439-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg439-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg439-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
oleg439-server	-	lustre-MDT0001	zfs:lustre-mdt2/mdt2
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg439-server	192.168.204.139@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-MDT0000
lustre-MDT0001
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
lustre-MDT0001
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
PASS 94 (8s)

== conf-sanity test 95: ldev should only allow one label filter ========================================================== 18:09:38 (1787609378)
----- /tmp/ldev.conf -----
oleg439-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg439-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg439-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg439-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
oleg439-server	-	lustre-MDT0001	zfs:lustre-mdt2/mdt2
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg439-server	192.168.204.139@tcp
--- END /tmp/nids ---
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
PASS 95 (10s)

== conf-sanity test 96: ldev returns hostname and backend fs correctly in command sub ========================================================== 18:09:47 (1787609387)
----- /tmp/ldev.conf -----
oleg439-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg439-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg439-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg439-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
oleg439-server	-	lustre-MDT0001	zfs:lustre-mdt2/mdt2
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg439-server	192.168.204.139@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
oleg439-server-zfs
oleg439-server-zfs
oleg439-server-zfs
oleg439-server-zfs
oleg439-server-zfs
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
oleg439-server-zfs
oleg439-server-zfs
oleg439-server-zfs
oleg439-server-zfs
oleg439-server-zfs
--- END OF EXPECTED OUTPUT ---
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
PASS 96 (11s)

== conf-sanity test 97: ldev returns correct ouput when querying based on role ========================================================== 18:09:58 (1787609398)
----- /tmp/ldev.conf -----
oleg439-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg439-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg439-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg439-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
oleg439-server	-	lustre-MDT0001	zfs:lustre-mdt2/mdt2
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg439-server	192.168.204.139@tcp
--- END /tmp/nids ---

MDT role
-- START OF LDEV OUTPUT --
lustre-MDT0000
lustre-MDT0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
lustre-MDT0001
--- END OF EXPECTED OUTPUT ---

OST role
-- START OF LDEV OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---

MGS role
-- START OF LDEV OUTPUT --
lustre-MGS0000
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MGS0000
--- END OF EXPECTED OUTPUT ---
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
PASS 97 (10s)

== conf-sanity test 98: Buffer-overflow check while parsing mount_opts ========================================================== 18:10:09 (1787609409)
start mds service on oleg439-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
setup single mount lustre success
arg[0] = mount.lustre
arg[1] = -o
arg[2] = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
arg[3] = -v
arg[4] = 192.168.204.139@tcp:/lustre
arg[5] = /mnt/lustre
source = 192.168.204.139@tcp:/lustre (192.168.204.139@tcp:/lustre), target = /mnt/lustre
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
PASS 98 (150s)

== conf-sanity test 99: Adding meta_bg option ============ 18:12:39 (1787609559)

 SKIP: conf-sanity test_99 ldiskfs only test
SKIP 99 (5s)

== conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 18:12:44 (1787609564)
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
oleg439-server: oleg439-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg439-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
setup single mount lustre success
lustre-MDT0000:
lustre-OST0000:
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
PASS 100 (226s)

== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:16:30 (1787609790)
start mds service on oleg439-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
 - open/close 69 (time 1787609909.88 total 10.30 last 6.70)
 - open/close 249 (time 1787609920.44 total 20.86 last 17.05)
 - open/close 371 (time 1787609930.53 total 30.96 last 12.08)
 - open/close 538 (time 1787609941.37 total 41.79 last 15.42)
 - open/close 663 (time 1787609951.38 total 51.81 last 12.48)
 - open/close 789 (time 1787609961.38 total 61.81 last 12.60)
 - open/close 910 (time 1787609971.75 total 72.18 last 11.67)
 - open/close 1007 (time 1787609981.76 total 82.18 last 9.70)
 - open/close 1088 (time 1787609992.29 total 92.72 last 7.69)
 - open/close 1126 (time 1787610003.81 total 104.24 last 3.30)
 - open/close 1282 (time 1787610014.38 total 114.81 last 14.76)
 - open/close 1509 (time 1787610025.32 total 125.74 last 20.76)
 - open/close 1628 (time 1787610035.32 total 135.74 last 11.90)
 - open/close 1699 (time 1787610045.74 total 146.16 last 6.81)
 - open/close 1841 (time 1787610055.75 total 156.17 last 14.19)
 - open/close 1924 (time 1787610066.24 total 166.66 last 7.91)
 - open/close 2034 (time 1787610076.28 total 176.70 last 10.96)
 - open/close 2163 (time 1787610086.71 total 187.13 last 12.37)
 - open/close 2251 (time 1787610096.72 total 197.14 last 8.79)
 - open/close 2399 (time 1787610107.66 total 208.08 last 13.53)
 - open/close 2498 (time 1787610117.66 total 218.09 last 9.90)
 - open/close 2624 (time 1787610128.55 total 228.97 last 11.57)
 - open/close 2723 (time 1787610141.61 total 242.04 last 7.58)
 - open/close 2986 (time 1787610151.64 total 252.07 last 26.22)
 - open/close 3187 (time 1787610163.19 total 263.61 last 17.41)
 - open/close 3250 (time 1787610174.25 total 274.67 last 5.69)
 - open/close 3459 (time 1787610184.28 total 284.71 last 20.83)
 - open/close 3566 (time 1787610194.30 total 294.72 last 10.69)
 - open/close 3656 (time 1787610204.85 total 305.27 last 8.53)
 - open/close 3736 (time 1787610215.05 total 315.48 last 7.84)
 - open/close 3875 (time 1787610226.13 total 326.55 last 12.55)
 - open/close 4046 (time 1787610236.15 total 336.57 last 17.06)
 - open/close 4275 (time 1787610246.22 total 346.65 last 22.73)
 - open/close 4370 (time 1787610256.75 total 357.18 last 9.02)
 - open/close 4540 (time 1787610267.31 total 367.73 last 16.10)
 - open/close 4617 (time 1787610277.54 total 377.97 last 7.52)
 - open/close 4689 (time 1787610287.66 total 388.09 last 7.11)
 - open/close 4776 (time 1787610300.21 total 400.63 last 6.93)
 - open/close 4831 (time 1787610310.95 total 411.37 last 5.12)
 - open/close 4934 (time 1787610320.96 total 421.38 last 10.29)
 - open/close 5118 (time 1787610333.70 total 434.12 last 14.44)
 - open/close 5289 (time 1787610344.11 total 444.53 last 16.43)
 - open/close 5372 (time 1787610354.11 total 454.54 last 8.30)
 - open/close 5424 (time 1787610364.60 total 465.02 last 4.96)
 - open/close 5627 (time 1787610374.63 total 475.05 last 20.25)
 - open/close 5758 (time 1787610384.64 total 485.06 last 13.08)
 - open/close 5895 (time 1787610395.00 total 495.42 last 13.22)
 - open/close 6101 (time 1787610405.92 total 506.34 last 18.87)
 - open/close 6240 (time 1787610415.96 total 516.39 last 13.84)
 - open/close 6474 (time 1787610426.78 total 527.21 last 21.62)
 - open/close 6571 (time 1787610437.30 total 537.72 last 9.23)
 - open/close 6725 (time 1787610449.96 total 550.38 last 12.16)
 - open/close 6792 (time 1787610460.01 total 560.43 last 6.66)
 - open/close 6956 (time 1787610470.50 total 570.92 last 15.64)
 - open/close 7019 (time 1787610481.46 total 581.89 last 5.75)
 - open/close 7285 (time 1787610491.46 total 591.89 last 26.59)
 - open/close 7375 (time 1787610501.54 total 601.96 last 8.93)
 - open/close 7582 (time 1787610511.54 total 611.97 last 20.69)
 - open/close 7775 (time 1787610521.57 total 621.99 last 19.25)
 - open/close 7984 (time 1787610531.65 total 632.08 last 20.72)
 - open/close 8188 (time 1787610542.25 total 642.67 last 19.26)
 - open/close 8295 (time 1787610553.27 total 653.70 last 9.70)
 - open/close 8491 (time 1787610563.63 total 664.05 last 18.93)
 - open/close 8627 (time 1787610573.67 total 674.09 last 13.54)
 - open/close 8762 (time 1787610583.93 total 684.35 last 13.16)
 - open/close 9026 (time 1787610594.48 total 694.91 last 25.01)
 - open/close 9250 (time 1787610604.51 total 704.93 last 22.35)
 - open/close 9456 (time 1787610615.48 total 715.91 last 18.77)
 - open/close 9618 (time 1787610625.49 total 725.92 last 16.18)
 - open/close 9752 (time 1787610636.47 total 736.89 last 12.21)
 - open/close 9892 (time 1787610646.50 total 746.92 last 13.96)
 - open/close 10000 (time 1787610651.09 total 751.52 last 23.49)
 - open/close 10148 (time 1787610661.80 total 762.22 last 13.83)
 - open/close 10261 (time 1787610674.31 total 774.74 last 9.03)
 - open/close 10393 (time 1787610684.46 total 784.88 last 13.02)
 - open/close 10439 (time 1787610694.95 total 795.38 last 4.38)
 - open/close 10673 (time 1787610705.84 total 806.26 last 21.50)
 - open/close 10911 (time 1787610718.64 total 819.07 last 18.58)
 - open/close 11056 (time 1787610729.25 total 829.67 last 13.67)
 - open/close 11231 (time 1787610739.86 total 840.28 last 16.49)
 - open/close 11408 (time 1787610749.88 total 850.30 last 17.67)
 - open/close 11631 (time 1787610759.91 total 860.34 last 22.22)
 - open/close 11882 (time 1787610770.88 total 871.31 last 22.88)
 - open/close 12023 (time 1787610785.33 total 885.75 last 9.76)
 - open/close 12163 (time 1787610795.83 total 896.25 last 13.33)
 - open/close 12341 (time 1787610806.06 total 906.48 last 17.40)
 - open/close 12526 (time 1787610816.07 total 916.49 last 18.48)
 - open/close 12747 (time 1787610826.12 total 926.54 last 21.99)
 - open/close 12801 (time 1787610836.15 total 936.58 last 5.38)
 - open/close 12871 (time 1787610848.25 total 948.67 last 5.79)
 - open/close 12926 (time 1787610858.35 total 958.77 last 5.45)
 - open/close 13048 (time 1787610868.35 total 968.78 last 12.19)
 - open/close 13259 (time 1787610878.37 total 978.80 last 21.06)
 - open/close 13502 (time 1787610889.37 total 989.80 last 22.09)
 - open/close 13725 (time 1787610899.38 total 999.80 last 22.29)
 - open/close 13851 (time 1787610909.69 total 1010.11 last 12.22)
 - open/close 14034 (time 1787610920.12 total 1020.54 last 17.54)
 - open/close 14161 (time 1787610930.44 total 1030.86 last 12.31)
 - open/close 14353 (time 1787610940.77 total 1041.20 last 18.58)
 - open/close 14447 (time 1787610951.17 total 1051.60 last 9.04)
 - open/close 14522 (time 1787610961.58 total 1062.01 last 7.21)
 - open/close 14578 (time 1787610971.72 total 1072.14 last 5.53)
 - open/close 14682 (time 1787610982.02 total 1082.45 last 10.09)
 - open/close 14786 (time 1787610992.05 total 1092.47 last 10.37)
 - open/close 14911 (time 1787611003.75 total 1104.18 last 10.68)
 - open/close 15038 (time 1787611013.76 total 1114.19 last 12.68)
 - open/close 15605 (time 1787611023.77 total 1124.19 last 56.67)
 - open/close 16204 (time 1787611033.77 total 1134.20 last 59.87)
 - open/close 16915 (time 1787611043.78 total 1144.20 last 71.07)
 - open/close 17531 (time 1787611053.78 total 1154.21 last 61.59)
 - open/close 18167 (time 1787611063.79 total 1164.21 last 63.56)
 - open/close 18792 (time 1787611073.79 total 1174.21 last 62.50)
 - open/close 19402 (time 1787611083.81 total 1184.23 last 60.90)
 - open/close 19955 (time 1787611093.81 total 1194.23 last 55.28)
 - open/close 20619 (time 1787611103.82 total 1204.24 last 66.35)
 - open/close 21259 (time 1787611113.82 total 1214.24 last 63.98)
 - open/close 21828 (time 1787611123.83 total 1224.25 last 56.85)
 - open/close 22319 (time 1787611133.85 total 1234.27 last 49.01)
 - open/close 22866 (time 1787611143.85 total 1244.27 last 54.69)
 - open/close 23452 (time 1787611153.85 total 1254.28 last 58.59)
 - open/close 24072 (time 1787611163.85 total 1264.28 last 61.99)
 - open/close 24674 (time 1787611173.86 total 1274.28 last 60.17)
 - open/close 25272 (time 1787611183.87 total 1284.30 last 59.70)
 - open/close 25877 (time 1787611193.88 total 1294.30 last 60.48)
 - open/close 26420 (time 1787611203.88 total 1304.31 last 54.26)
 - open/close 27060 (time 1787611213.89 total 1314.32 last 63.94)
 - open/close 27650 (time 1787611223.90 total 1324.33 last 58.95)
 - open/close 28307 (time 1787611233.91 total 1334.33 last 65.68)
 - open/close 28939 (time 1787611243.92 total 1344.34 last 63.13)
 - open/close 29515 (time 1787611253.94 total 1354.36 last 57.47)
 - open/close 30000 (time 1787611261.29 total 1361.72 last 65.95)
 - open/close 30623 (time 1787611271.30 total 1371.73 last 62.23)
 - open/close 31335 (time 1787611281.31 total 1381.73 last 71.18)
 - open/close 32018 (time 1787611291.32 total 1391.74 last 68.21)
 - open/close 32690 (time 1787611301.33 total 1401.75 last 67.15)
 - open/close 33350 (time 1787611311.33 total 1411.75 last 66.00)
 - open/close 33949 (time 1787611321.34 total 1421.76 last 59.82)
 - open/close 34453 (time 1787611331.36 total 1431.78 last 50.30)
 - open/close 35126 (time 1787611341.37 total 1441.79 last 67.26)
 - open/close 35748 (time 1787611351.37 total 1451.79 last 62.17)
 - open/close 36238 (time 1787611361.39 total 1461.81 last 48.90)
 - open/close 36696 (time 1787611371.40 total 1471.82 last 45.76)
 - open/close 37283 (time 1787611381.41 total 1481.83 last 58.67)
 - open/close 37916 (time 1787611391.41 total 1491.84 last 63.25)
 - open/close 38528 (time 1787611401.42 total 1501.85 last 61.14)
 - open/close 39222 (time 1787611411.43 total 1511.85 last 69.35)
 - open/close 39893 (time 1787611421.44 total 1521.86 last 67.04)
 - open/close 40437 (time 1787611431.46 total 1531.89 last 54.27)
 - open/close 41066 (time 1787611441.46 total 1541.89 last 62.88)
 - open/close 41641 (time 1787611451.47 total 1551.89 last 57.47)
 - open/close 42203 (time 1787611461.51 total 1561.93 last 56.00)
 - open/close 42761 (time 1787611471.51 total 1571.94 last 55.77)
 - open/close 43421 (time 1787611481.55 total 1581.97 last 65.75)
 - open/close 44038 (time 1787611491.55 total 1591.98 last 61.67)
 - open/close 44622 (time 1787611501.56 total 1601.98 last 58.38)
 - open/close 45134 (time 1787611511.57 total 1611.99 last 51.17)
 - open/close 45639 (time 1787611521.57 total 1621.99 last 50.48)
 - open/close 46142 (time 1787611531.58 total 1632.01 last 50.22)
 - open/close 46688 (time 1787611541.60 total 1642.02 last 54.54)
 - open/close 47262 (time 1787611551.65 total 1652.08 last 57.08)
 - open/close 47810 (time 1787611561.65 total 1662.08 last 54.79)
 - open/close 48389 (time 1787611571.66 total 1672.08 last 57.89)
 - open/close 48922 (time 1787611581.67 total 1682.10 last 53.20)
 - open/close 49434 (time 1787611591.68 total 1692.11 last 51.15)
total: 50000 open/close in 1701.42 seconds: 29.39 ops/second
 - unlinked 0 (time 1787611606 ; total 0 ; last 0)
 - unlinked 10000 (time 1787611783 ; total 177 ; last 177)
 - unlinked 20000 (time 1787611951 ; total 345 ; last 168)
 - unlinked 30000 (time 1787612122 ; total 516 ; last 171)
 - unlinked 40000 (time 1787612292 ; total 686 ; last 170)
total: 50000 unlinks in 854 seconds: 58.548008 unlinks/second
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
PASS 101a (2728s)

== conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 19:01:59 (1787612519)
start mds service on oleg439-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
fail_loc=0x80002107
fail_val=20
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff97f2075c2800.ost_server_uuid 50
osc.lustre-OST0000-osc-ffff97f2075c2800.ost_server_uuid in FULL state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
PASS 101b (234s)

 SKIP: conf-sanity test_102 skipping excluded test 102

== conf-sanity test 103: rename filesystem name ========== 19:05:56 (1787612756)
Checking servers environments
Checking clients oleg439-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Starting client oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Started clients oleg439-client.virtnet: 
192.168.204.139@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff97f205551000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff97f205551000.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 2s: want 'procname_uid' got 'procname_uid'
disable quota as required
oleg439-server: Pool lustre.pool1 created
oleg439-server: Pool lustre.lustre created
oleg439-server: OST lustre-OST0000_UUID added to pool lustre.lustre
Waiting 90s for 'lustre-OST0000_UUID'
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:)
Stopping client oleg439-client.virtnet /mnt/lustre opts:
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
rename lustre to mylustre
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-mdt1/mdt1
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20

   Permanent disk data:
Device:     lustre-mdt1/mdt1
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20
  lustre:svname=mylustre-MDT0000
cmd: zfs set mountpoint=/tmp/mntQ895sw lustre-mdt1/mdt1 && zfs set canmount=on lustre-mdt1/mdt1 && zfs mount lustre-mdt1/mdt1
cmd: zfs umount lustre-mdt1/mdt1 && zfs set canmount=off lustre-mdt1/mdt1 && zfs set mountpoint=none lustre-mdt1/mdt1
Writing lustre-mdt1/mdt1 properties
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:sys.timeout=20
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=mylustre
  lustre:svname=mylustre-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-mdt2/mdt2
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mgsnode=192.168.204.139@tcp

   Permanent disk data:
Device:     lustre-mdt2/mdt2
Target:     mylustre-MDT0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mgsnode=192.168.204.139@tcp
  lustre:svname=mylustre-MDT0001
cmd: zfs set mountpoint=/tmp/mntVhCaXO lustre-mdt2/mdt2 && zfs set canmount=on lustre-mdt2/mdt2 && zfs mount lustre-mdt2/mdt2
cmd: zfs umount lustre-mdt2/mdt2 && zfs set canmount=off lustre-mdt2/mdt2 && zfs set mountpoint=none lustre-mdt2/mdt2
Writing lustre-mdt2/mdt2 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:mgsnode=192.168.204.139@tcp
  lustre:version=1
  lustre:flags=1
  lustre:index=1
  lustre:fsname=mylustre
  lustre:svname=mylustre-MDT0001
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.139@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.139@tcp autodegrade=on
  lustre:svname=mylustre-OST0000
cmd: zfs set mountpoint=/tmp/mntWwpVUW lustre-ost1/ost1 && zfs set canmount=on lustre-ost1/ost1 && zfs mount lustre-ost1/ost1
cmd: zfs umount lustre-ost1/ost1 && zfs set canmount=off lustre-ost1/ost1 && zfs set mountpoint=none lustre-ost1/ost1
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.204.139@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=mylustre
  lustre:svname=mylustre-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost2/ost2
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.139@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost2/ost2
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.139@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=mylustre-OST0001
cmd: zfs set mountpoint=/tmp/mntCvC4nk lustre-ost2/ost2 && zfs set canmount=on lustre-ost2/ost2 && zfs mount lustre-ost2/ost2
cmd: zfs umount lustre-ost2/ost2 && zfs set canmount=off lustre-ost2/ost2 && zfs set mountpoint=none lustre-ost2/ost2
Writing lustre-ost2/ost2 properties
  lustre:mgsnode=192.168.204.139@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=mylustre
  lustre:svname=mylustre-OST0001
Checking servers environments
Checking clients oleg439-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  mylustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started mylustre-MDT0000
Start mds2: mount -t lustre -o localrecov  mylustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started mylustre-MDT0001
Start ost1: mount -t lustre -o localrecov  mylustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-mylustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started mylustre-OST0000
Start ost2: mount -t lustre -o localrecov  mylustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-mylustre-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started mylustre-OST0001
mount mylustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/mylustre /mnt/lustre
Starting client oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/mylustre /mnt/lustre
Started clients oleg439-client.virtnet: 
192.168.204.139@tcp:/mylustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.mylustre-OST0000-osc-ffff97f22ef26800.idle_timeout=debug
osc.mylustre-OST0001-osc-ffff97f22ef26800.idle_timeout=debug
disable quota as required
  File: /mnt/lustre/d103.conf-sanity/test-framework.sh
  Size: 317150    	Blocks: 1027       IO Block: 4194304 regular file
Device: c3aa56ceh/3282720462d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-08-24 19:07:55.000000000 -0400
Modify: 2026-08-24 19:07:55.000000000 -0400
Change: 2026-08-24 19:07:55.000000000 -0400
 Birth: 2026-08-24 19:07:56.000000000 -0400
Pool: mylustre.pool1
Pool: mylustre.lustre
mylustre-OST0000_UUID
mylustre-OST0000_UUID
oleg439-server: OST mylustre-OST0001_UUID added to pool mylustre.lustre
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:)
Stopping client oleg439-client.virtnet /mnt/lustre opts:
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
rename mylustre to tfs
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-mdt1/mdt1
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20

   Permanent disk data:
Device:     mylustre-mdt1/mdt1
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20
  lustre:svname=tfs-MDT0000
cmd: zfs set mountpoint=/tmp/mntNRFfO6 mylustre-mdt1/mdt1 && zfs set canmount=on mylustre-mdt1/mdt1 && zfs mount mylustre-mdt1/mdt1
cmd: zfs umount mylustre-mdt1/mdt1 && zfs set canmount=off mylustre-mdt1/mdt1 && zfs set mountpoint=none mylustre-mdt1/mdt1
Writing mylustre-mdt1/mdt1 properties
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:sys.timeout=20
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=tfs
  lustre:svname=tfs-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-mdt2/mdt2
Target:     mylustre-MDT0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mgsnode=192.168.204.139@tcp

   Permanent disk data:
Device:     mylustre-mdt2/mdt2
Target:     tfs-MDT0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mgsnode=192.168.204.139@tcp
  lustre:svname=tfs-MDT0001
cmd: zfs set mountpoint=/tmp/mnte4Mwug mylustre-mdt2/mdt2 && zfs set canmount=on mylustre-mdt2/mdt2 && zfs mount mylustre-mdt2/mdt2
cmd: zfs umount mylustre-mdt2/mdt2 && zfs set canmount=off mylustre-mdt2/mdt2 && zfs set mountpoint=none mylustre-mdt2/mdt2
Writing mylustre-mdt2/mdt2 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:mgsnode=192.168.204.139@tcp
  lustre:version=1
  lustre:flags=1
  lustre:index=1
  lustre:fsname=tfs
  lustre:svname=tfs-MDT0001
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-ost1/ost1
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.139@tcp

   Permanent disk data:
Device:     mylustre-ost1/ost1
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.139@tcp autodegrade=on
  lustre:svname=tfs-OST0000
cmd: zfs set mountpoint=/tmp/mntRWrr0V mylustre-ost1/ost1 && zfs set canmount=on mylustre-ost1/ost1 && zfs mount mylustre-ost1/ost1
cmd: zfs umount mylustre-ost1/ost1 && zfs set canmount=off mylustre-ost1/ost1 && zfs set mountpoint=none mylustre-ost1/ost1
Writing mylustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.204.139@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=tfs
  lustre:svname=tfs-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-ost2/ost2
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.139@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     mylustre-ost2/ost2
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.139@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=tfs-OST0001
cmd: zfs set mountpoint=/tmp/mntMAltxG mylustre-ost2/ost2 && zfs set canmount=on mylustre-ost2/ost2 && zfs mount mylustre-ost2/ost2
cmd: zfs umount mylustre-ost2/ost2 && zfs set canmount=off mylustre-ost2/ost2 && zfs set mountpoint=none mylustre-ost2/ost2
Writing mylustre-ost2/ost2 properties
  lustre:mgsnode=192.168.204.139@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=tfs
  lustre:svname=tfs-OST0001
Checking servers environments
Checking clients oleg439-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  tfs-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started tfs-MDT0000
Start mds2: mount -t lustre -o localrecov  tfs-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started tfs-MDT0001
Start ost1: mount -t lustre -o localrecov  tfs-ost1/ost1 /mnt/lustre-ost1
seq.cli-tfs-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started tfs-OST0000
Start ost2: mount -t lustre -o localrecov  tfs-ost2/ost2 /mnt/lustre-ost2
seq.cli-tfs-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started tfs-OST0001
mount tfs  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/tfs /mnt/lustre
Starting client oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/tfs /mnt/lustre
Started clients oleg439-client.virtnet: 
192.168.204.139@tcp:/tfs on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.tfs-OST0000-osc-ffff97f209922000.idle_timeout=debug
osc.tfs-OST0001-osc-ffff97f209922000.idle_timeout=debug
disable quota as required
  File: /mnt/lustre/d103.conf-sanity/test-framework.sh
  Size: 317150    	Blocks: 1027       IO Block: 4194304 regular file
Device: 32e2fa5ah/853736026d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-08-24 19:07:55.000000000 -0400
Modify: 2026-08-24 19:07:55.000000000 -0400
Change: 2026-08-24 19:07:55.000000000 -0400
 Birth: 2026-08-24 19:07:56.000000000 -0400
Pool: tfs.pool1
Pool: tfs.lustre
tfs-OST0000_UUID
tfs-OST0001_UUID
tfs-OST0000_UUID
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:)
Stopping client oleg439-client.virtnet /mnt/lustre opts:
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
rename tfs to lustre
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-mdt1/mdt1
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20

   Permanent disk data:
Device:     tfs-mdt1/mdt1
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20
  lustre:svname=lustre-MDT0000
cmd: zfs set mountpoint=/tmp/mntGR3xcu tfs-mdt1/mdt1 && zfs set canmount=on tfs-mdt1/mdt1 && zfs mount tfs-mdt1/mdt1
cmd: zfs umount tfs-mdt1/mdt1 && zfs set canmount=off tfs-mdt1/mdt1 && zfs set mountpoint=none tfs-mdt1/mdt1
Writing tfs-mdt1/mdt1 properties
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:sys.timeout=20
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-mdt2/mdt2
Target:     tfs-MDT0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mgsnode=192.168.204.139@tcp

   Permanent disk data:
Device:     tfs-mdt2/mdt2
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mgsnode=192.168.204.139@tcp
  lustre:svname=lustre-MDT0001
cmd: zfs set mountpoint=/tmp/mntcbM5lT tfs-mdt2/mdt2 && zfs set canmount=on tfs-mdt2/mdt2 && zfs mount tfs-mdt2/mdt2
cmd: zfs umount tfs-mdt2/mdt2 && zfs set canmount=off tfs-mdt2/mdt2 && zfs set mountpoint=none tfs-mdt2/mdt2
Writing tfs-mdt2/mdt2 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:mgsnode=192.168.204.139@tcp
  lustre:version=1
  lustre:flags=1
  lustre:index=1
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0001
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-ost1/ost1
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.139@tcp

   Permanent disk data:
Device:     tfs-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.139@tcp autodegrade=on
  lustre:svname=lustre-OST0000
cmd: zfs set mountpoint=/tmp/mntMiulRk tfs-ost1/ost1 && zfs set canmount=on tfs-ost1/ost1 && zfs mount tfs-ost1/ost1
cmd: zfs umount tfs-ost1/ost1 && zfs set canmount=off tfs-ost1/ost1 && zfs set mountpoint=none tfs-ost1/ost1
Writing tfs-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.204.139@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-ost2/ost2
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.139@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     tfs-ost2/ost2
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.139@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0001
cmd: zfs set mountpoint=/tmp/mntxZMv1T tfs-ost2/ost2 && zfs set canmount=on tfs-ost2/ost2 && zfs mount tfs-ost2/ost2
cmd: zfs umount tfs-ost2/ost2 && zfs set canmount=off tfs-ost2/ost2 && zfs set mountpoint=none tfs-ost2/ost2
Writing tfs-ost2/ost2 properties
  lustre:mgsnode=192.168.204.139@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=lustre
  lustre:svname=lustre-OST0001
Checking servers environments
Checking clients oleg439-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Starting client oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
Started clients oleg439-client.virtnet: 
192.168.204.139@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff97f205554000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff97f205554000.idle_timeout=debug
disable quota as required
PASS 103 (611s)

== conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 19:16:07 (1787613367)
mkfs.lustre --mountfsopts: user_xattr,recovery_time_soft=222
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg439-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
oleg439-server: oleg439-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
mount opts: -o localrecov,recovery_time_soft=188
Start mds1: mount -t lustre -o localrecov,recovery_time_soft=188  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov,recovery_time_soft=188  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
found wanted mdt.lustre-MDT0000.recovery_time_soft=188
PASS 104a (213s)

== conf-sanity test 104b: Mount uses last flock argument ========================================================== 19:19:40 (1787613580)
mount lustre with opts flock,localflock on /mnt/lustre3.....
Starting client: oleg439-client.virtnet:  -o flock,localflock 192.168.204.139@tcp:/lustre /mnt/lustre3
192.168.204.139@tcp:/lustre on /mnt/lustre3 type lustre (rw,localflock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,nouser_xattr,verbose)
umount lustre on /mnt/lustre3.....
Stopping client oleg439-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock on /mnt/lustre3.....
Starting client: oleg439-client.virtnet:  -o localflock,flock 192.168.204.139@tcp:/lustre /mnt/lustre3
192.168.204.139@tcp:/lustre on /mnt/lustre3 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,nouser_xattr,verbose)
umount lustre on /mnt/lustre3.....
Stopping client oleg439-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock,noflock on /mnt/lustre3.....
Starting client: oleg439-client.virtnet:  -o localflock,flock,noflock 192.168.204.139@tcp:/lustre /mnt/lustre3
umount lustre on /mnt/lustre3.....
Stopping client oleg439-client.virtnet /mnt/lustre3 (opts:)
PASS 104b (19s)

== conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 19:20:00 (1787613600)
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: lustre_rmmod: wait 42s for 6 OBD devices
oleg439-server: lustre_rmmod: OBD devices after 60s:
oleg439-server: 
oleg439-server:   2 UP mgc MGC192.168.204.139@tcp 3fa24ad7-68e1-4afd-8dae-128404db07c9 5
oleg439-server:  17 UP ost OSS OSS_uuid 3
oleg439-server:  23 UP osd-zfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5
oleg439-server:  24 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 3
oleg439-server:  25 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5
oleg439-server:  26 UP lwp lustre-MDT0001-lwp-OST0001 lustre-MDT0001-lwp-OST0001_UUID 5
oleg439-server: rmmod: ERROR: Module mdd is in use
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server
oleg439-server: oleg439-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg439-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg439-client.virtnet:  -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre
touch: cannot touch '/tmp/d105.conf-sanity/f105.conf-sanity': Read-only file system
umount lustre on /mnt/lustre.....
Stopping client oleg439-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
PASS 105 (353s)

 SKIP: conf-sanity test_106 skipping SLOW test 106

== conf-sanity test 107: Unknown config param should not fail target mounting ========================================================== 19:25:55 (1787613955)
start mds service on oleg439-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg439-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
start ost1 service on oleg439-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1
Started lustre-OST0000
oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
umount lustre on /mnt/lustre.....
stop ost1 service on oleg439-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server
stop mds service on oleg439-server
stop mds service on oleg439-server

unloading modules via unload_modules_local on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing unload_modules_local
oleg439-server: 
modules unloaded.
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2
PASS 107 (282s)

 SKIP: conf-sanity test_108a skipping ALWAYS excluded test 108a

== conf-sanity test 108b: migrate from ZFS to ldiskfs ==== 19:30:39 (1787614239)

 SKIP: conf-sanity test_108b ldiskfs only test
SKIP 108b (4s)

== conf-sanity test 109a: test lctl clear_conf fsname ==== 19:30:44 (1787614244)

 SKIP: conf-sanity test_109a LU-8727: no implementation for ZFS
SKIP 109a (3s)

== conf-sanity test 109b: test lctl clear_conf one config ========================================================== 19:30:47 (1787614247)

 SKIP: conf-sanity test_109b LU-8727: no implementation for ZFS
SKIP 109b (4s)

== conf-sanity test 110: Adding large_dir with 3-level htree ========================================================== 19:30:51 (1787614251)

 SKIP: conf-sanity test_110 Only applicable to ldiskfs-based MDTs
SKIP 110 (3s)

 SKIP: conf-sanity test_111 skipping SLOW test 111

 SKIP: conf-sanity test_112a skipping ALWAYS excluded test 112a

 SKIP: conf-sanity test_112b skipping ALWAYS excluded test 112b

 SKIP: conf-sanity test_113 skipping ALWAYS excluded test 113

 SKIP: conf-sanity test_114 skipping SLOW test 114

 SKIP: conf-sanity test_115 skipping excluded test 115

== conf-sanity test 116: big size MDT support ============ 19:31:06 (1787614266)

 SKIP: conf-sanity test_116 ldiskfs only test
SKIP 116 (4s)

 SKIP: conf-sanity test_117 skipping ALWAYS excluded test 117

 SKIP: conf-sanity test_119 skipping ALWAYS excluded test 119

== conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 19:31:14 (1787614274)

 SKIP: conf-sanity test_120 ldiskfs only test
SKIP 120 (4s)
Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2
oleg439-server: oleg439-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg439-server'
oleg439-server: oleg439-server.virtnet: executing load_modules_local
oleg439-server: Loading modules from /home/green/git/lustre-release/lustre
oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg439-server: detected 4 online CPUs by sysfs
oleg439-server: MODOPTS_LIBCFS=
oleg439-server: Force libcfs to create 2 CPU partitions
oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
== conf-sanity test complete, duration 8492 sec ========== 19:33:02 (1787614382)
=== conf-sanity: start cleanup 19:33:05 (1787614385) ===
=== conf-sanity: finish cleanup 19:33:09 (1787614389) ===