-----============= acceptance-small: conf-sanity ============----- Sat Aug 15 08:37:23 EDT 2026
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg321-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex
- see CLIENT_VERSION > v2_15_63-53-g1a6ef725c2 (34682625 > 34553653) for LU-16938, go 81
excepting tests: 102 106 115 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a
skipping tests SLOW=no: 45 69 106 111 114
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg321-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
oleg321-server: oleg321-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg321-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg321-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg321-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

== conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 08:40:47 (1786797647)
start mds service on oleg321-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Change MGS params
max_dirty_mb: 462
new_max_dirty_mb: 452
Waiting 90s for '452'
Updated after 2s: want '452' got '452'
452
Check the value is stored after remount
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:)
Stopping client oleg321-client.virtnet /mnt/lustre opts:
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
Checking servers environments
Checking clients oleg321-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Starting client oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Started clients oleg321-client.virtnet: 
192.168.203.121@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff96878598b000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff96878598b000.idle_timeout=debug
disable quota as required
Change OST params
client_cache_count: 128
new_client_cache_count: 256
Waiting 90s for '256'
Updated after 2s: want '256' got '256'
256
Check the value is stored after remount
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:)
Stopping client oleg321-client.virtnet /mnt/lustre opts:
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
Checking servers environments
Checking clients oleg321-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Starting client oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Started clients oleg321-client.virtnet: 
192.168.203.121@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff96878942d000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff96878942d000.idle_timeout=debug
disable quota as required
256
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:)
Stopping client oleg321-client.virtnet /mnt/lustre opts:
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
PASS 76a (343s)

== conf-sanity test 76b: verify params log setup correctly ========================================================== 08:46:30 (1786797990)
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:)
Checking servers environments
Checking clients oleg321-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Starting client oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Started clients oleg321-client.virtnet: 
192.168.203.121@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9687915b4800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9687915b4800.idle_timeout=debug
disable quota as required
mgs.MGS.live.params=
fsname: params
flags: 0x20     gen: 1

Secure RPC Config Rules:

imperative_recovery_state:
    state: startup
    nonir_clients: 0
    nidtbl_version: 2
    notify_duration_total: 0.000000000
    notify_duation_max: 0.000000000
    notify_count: 0
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:)
Stopping client oleg321-client.virtnet /mnt/lustre opts:
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
PASS 76b (136s)

== conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 08:48:46 (1786798126)
Checking servers environments
Checking clients oleg321-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Starting client oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Started clients oleg321-client.virtnet: 
192.168.203.121@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9687830f8000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9687830f8000.idle_timeout=debug
disable quota as required
Change changelog_mask
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Check the value is stored after mds remount
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
start mds service on oleg321-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:)
Stopping client oleg321-client.virtnet /mnt/lustre opts:
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
PASS 76c (211s)

== conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 08:52:17 (1786798337)
Checking servers environments
Checking clients oleg321-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Starting client oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Started clients oleg321-client.virtnet: 
192.168.203.121@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff96878955d000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff96878955d000.idle_timeout=debug
disable quota as required
lctl set_param -P llite.*.xattr_cache=0
Waiting 90s for '0'
Check llite.$inst.xattr_cache on client /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Check llite.$inst.xattr_cache on the new client /mnt/lustre2
mount lustre  on /mnt/lustre2.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre2
umount lustre on /mnt/lustre2.....
Stopping client oleg321-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:)
Stopping client oleg321-client.virtnet /mnt/lustre opts:
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
oleg321-server: No device found for name MGS: Invalid argument
oleg321-server: This command must be run on the MGS.
oleg321-server: error: executing set_param: No such device
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 19
Reader error: 'No devices found' at 0
PASS 76d (147s)

== conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 08:54:44 (1786798484)
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1

   Permanent disk data:
Device:     lustre-mdt1_2/mdt1_2
Target:     test1234:MDT0000
Index:      0
Lustre FS:  test1234
Mount type: zfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1_2/mdt1_2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1_2/mdt1_2 properties
  lustre:mgsnode=192.168.203.121@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=101
  lustre:index=0
  lustre:fsname=test1234
  lustre:svname=test1234:MDT0000
Start fs2mds: mount -t lustre -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1_2/mdt1_2
Started test1234-MDT0000

   Permanent disk data:
Device:     lustre-ost1_2/ost1_2
Target:     test1234:OST0000
Index:      0
Lustre FS:  test1234
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=0.0.0.0@tcp,192.168.203.121@tcp,192.168.203.121@tcp:192.168.203.121@tcp:192.168.203.121@tcp sys.timeout=20 failover.node=1.2.3.4@tcp,4.3.2.1@tcp autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1_2/ost1_2
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1_2/ost1_2 properties
  lustre:mgsnode=0.0.0.0@tcp,192.168.203.121@tcp,192.168.203.121@tcp:192.168.203.121@tcp:192.168.203.121@tcp
  lustre:sys.timeout=20
  lustre:failover.node=1.2.3.4@tcp,4.3.2.1@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=test1234
  lustre:svname=test1234:OST0000
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1849: OST_INDICES: bad array subscript
Start fs2ost: mount -t lustre -o localrecov  lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost
seq.cli-test1234-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-ost1_2/ost1_2
Started test1234-OST0000
setup single mount lustre success
umount /mnt/lustre2 ...
stopping fs2mds ...
Stopping /mnt/lustre-fs2mds (opts:-f) on oleg321-server
stopping fs2ost ...
Stopping /mnt/lustre-fs2ost (opts:-f) on oleg321-server
PASS 77 (78s)

== conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 08:56:02 (1786798562)

 SKIP: conf-sanity test_78 ldiskfs only test
SKIP 78 (4s)

== conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 08:56:06 (1786798566)
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: 
oleg321-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg321-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 22
oleg321-server: 
oleg321-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg321-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 22
oleg321-server: 
oleg321-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg321-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 22
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:-f)
oleg321-server: oleg321-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg321-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg321-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg321-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
PASS 79 (215s)

== conf-sanity test 80: mgc import reconnect race ======== 08:59:42 (1786798782)
start mds service on oleg321-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
fail_val=10
fail_loc=0x906
fail_val=10
fail_loc=0x906
start ost2 service on oleg321-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0
fail_loc=0
stop ost2 service on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
PASS 80 (164s)

== conf-sanity test 81: sparse OST indexing ============== 09:02:26 (1786798946)

 SKIP: conf-sanity test_81 needs >= 3 OSTs
SKIP 81 (3s)

== conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 09:02:29 (1786798949)

 SKIP: conf-sanity test_82a needs >= 3 OSTs
SKIP 82a (3s)

== conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 09:02:32 (1786798952)

 SKIP: conf-sanity test_82b needs >= 4 OSTs
SKIP 82b (3s)

== conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 09:02:35 (1786798955)

 SKIP: conf-sanity test_82c needs >= 4 OSTs
SKIP 82c (3s)

== conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 09:02:38 (1786798958)

 SKIP: conf-sanity test_83 ldiskfs only test
SKIP 83 (4s)

== conf-sanity test 84: check recovery_hard_time ========= 09:02:42 (1786798962)
start mds service on oleg321-server
start mds service on oleg321-server
Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg321-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg321-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
start ost2 service on oleg321-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0001
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg321-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg321-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
recovery_time=60, timeout=20, wrap_up=5
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre2.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre2
UUID                   1K-blocks        Used   Available Use%  Mounted on
lustre-MDT0000_UUID       383616        2048      379520   1%  /mnt/lustre[MDT:0] 
lustre-MDT0001_UUID       383616        1920      379648   1%  /mnt/lustre[MDT:1] 
lustre-OST0000_UUID       382976        2048      378880   1%  /mnt/lustre[OST:0] 
lustre-OST0001_UUID       382976        2048      378880   1%  /mnt/lustre[OST:1] 

filesystem_summary:       765952        4096      757760   1%  /mnt/lustre 

 - open/close 476 (time 1786799084.42 total 10.02 last 47.51)
total: 1000 open/close in 19.35 seconds: 51.67 ops/second
fail_loc=0x20000709
fail_val=5
Failing mds1 on oleg321-server
Stopping /mnt/lustre-mds1 (opts:) on oleg321-server
09:05:01 (1786799101) shut down
facet: mds1 facet_host: oleg321-server facet_failover_host: oleg321-server
Failover mds1 to oleg321-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
09:05:23 (1786799123) targets are mounted
09:05:23 (1786799123) facet_failover done
oleg321-client: error: invalid path '/mnt/lustre': Input/output error
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 5
recovery status
status: COMPLETE
recovery_start: 1786799119
recovery_duration: 60
completed_clients: 2/3
replayed_requests: 182
last_transno: 12884902074
VBR: DISABLED
IR: DISABLED
fail_loc=0
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
umount lustre on /mnt/lustre2.....
Stopping client oleg321-client.virtnet /mnt/lustre2 (opts:)
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop ost2 service on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
PASS 84 (264s)

== conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 09:07:06 (1786799226)
fail_loc=0x197
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
PASS 85 (79s)

== conf-sanity test 86: Replacing mkfs.lustre -G option == 09:08:25 (1786799305)

 SKIP: conf-sanity test_86 LU-6442: no such mkfs params for ZFS OSTs
SKIP 86 (4s)

== conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 09:08:29 (1786799309)

 SKIP: conf-sanity test_87 ldiskfs only test
SKIP 87 (3s)

== conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 09:08:32 (1786799312)

 SKIP: conf-sanity test_88a LU-6662: no implementation for ZFS
SKIP 88a (4s)

== conf-sanity test 88b: check the default mount options can be overridden ========================================================== 09:08:36 (1786799316)

 SKIP: conf-sanity test_88b LU-6662: no implementation for ZFS
SKIP 88b (3s)

== conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 09:08:40 (1786799320)
Failing mds1 on oleg321-server
09:08:44 (1786799324) shut down
facet: mds1 facet_host: oleg321-server facet_failover_host: oleg321-server
Failover mds1 to oleg321-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
09:09:11 (1786799351) targets are mounted
09:09:11 (1786799351) facet_failover done
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 95
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 95
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
tunefs --param failover.node=192.0.2.254@tcp0
tunefs --param failover.node=192.0.2.255@tcp0
tunefs --erase-param failover.node
tunefs --erase-params
tunefs --param failover.node=192.0.2.254@tcp0 --erase-params
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:-f)
oleg321-server: oleg321-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg321-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg321-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg321-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
PASS 89 (279s)

== conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 09:13:19 (1786799599)
start mds service on oleg321-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 8 files ...
fail_loc=0x159
launch 6 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-7 has perms 0600 OK
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-8 has perms 0644 OK
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
PASS 90a (183s)

== conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ========================================================== 09:16:23 (1786799783)
start mds service on oleg321-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff968791b1c000.max_mod_rpcs_in_flight=1
max_mod_rpcs_in_flight set to 1
creating 2 files ...
fail_loc=0x159
launch 0 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-1 has perms 0600 OK
fail_loc=0x159
launch 1 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-2 has perms 0644 OK
mdc.lustre-MDT0001-mdc-ffff968791b1c000.max_mod_rpcs_in_flight=5
max_mod_rpcs_in_flight set to 5
creating 6 files ...
fail_loc=0x159
launch 4 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-5 has perms 0600 OK
fail_loc=0x159
launch 5 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-6 has perms 0644 OK
mdt_max_mod_rpcs_in_flight is 8
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff9687830f8800.max_rpcs_in_flight=17
mdc.lustre-MDT0000-mdc-ffff9687830f8800.max_mod_rpcs_in_flight=16
max_mod_rpcs_in_flight set to 16
creating 17 files ...
fail_loc=0x159
launch 15 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-16 has perms 0600 OK
fail_loc=0x159
launch 16 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-17 has perms 0644 OK
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
PASS 90b (239s)

== conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ========================================================== 09:20:21 (1786800021)
start mds service on oleg321-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
max_rpcs_in_flight is 8
MDC max_mod_rpcs_in_flight is 7
mdt_max_mod_rpcs_in_flight is 8
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mdc.lustre-MDT0000-mdc-ffff968789b8a000.max_mod_rpcs_in_flight=8
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff968783057800.max_rpcs_in_flight=10
error: set_param: setting /sys/fs/lustre/mdc/lustre-MDT0000-mdc-ffff968783057800/max_mod_rpcs_in_flight=9: Numerical result out of range
error: set_param: setting 'mdc/lustre-MDT0000-mdc-*/max_mod_rpcs_in_flight'='9': Numerical result out of range
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
PASS 90c (139s)

== conf-sanity test 90d: check one close RPC is allowed above max_mod_rpcs_in_flight ========================================================== 09:22:40 (1786800160)
start mds service on oleg321-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 7 files ...
multiop /mnt/lustre/d90d.conf-sanity/file-close vO_c
TMPPIPE=/tmp/multiop_open_wait_pipe.8414
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional close in parallel ...
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
PASS 90d (139s)

== conf-sanity test 91: evict-by-nid support ============= 09:24:59 (1786800299)
start mds service on oleg321-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
setup single mount lustre success
list nids on mdt:
mdt.lustre-MDT0000.exports.0@lo
mdt.lustre-MDT0000.exports.192.168.203.21@tcp
mdt.lustre-MDT0000.exports.clear
mdt.lustre-MDT0001.exports.0@lo
mdt.lustre-MDT0001.exports.192.168.203.21@tcp
mdt.lustre-MDT0001.exports.clear
uuid from 192\.168\.203\.21@tcp:
mdt.lustre-MDT0000.exports.192.168.203.21@tcp.uuid=7e463549-97cf-492d-be0e-74be8817b272
mdt.lustre-MDT0001.exports.192.168.203.21@tcp.uuid=7e463549-97cf-492d-be0e-74be8817b272
manual umount lustre on /mnt/lustre....
evict 192\.168\.203\.21@tcp
oleg321-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.203.21@tcp/uuid': No such device
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 19
oleg321-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.203.21@tcp/uuid': No such device
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 19
oleg321-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.203.21@tcp/uuid': No such device
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 19
oleg321-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.203.21@tcp/uuid': No such device
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 19
umount lustre on /mnt/lustre.....
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
PASS 91 (187s)

== conf-sanity test 92: ldev returns MGS NID correctly in command substitution ========================================================== 09:28:06 (1786800486)
Host is oleg321-client.virtnet
----- /tmp/ldev.conf -----
oleg321-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg321-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg321-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg321-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
oleg321-server	-	lustre-MDT0001	zfs:lustre-mdt2/mdt2
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg321-server	192.168.203.121@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.203.121@tcp
lustre-OST0001: 192.168.203.121@tcp
lustre-MDT0001: 192.168.203.121@tcp
lustre-MGS0000: 192.168.203.121@tcp
lustre-MDT0000: 192.168.203.121@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-MGS0000: 192.168.203.121@tcp
lustre-OST0001: 192.168.203.121@tcp
lustre-OST0000: 192.168.203.121@tcp
lustre-MDT0001: 192.168.203.121@tcp
lustre-MDT0000: 192.168.203.121@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.203.121@tcp
lustre-MDT0001: 192.168.203.121@tcp
lustre-MDT0000: 192.168.203.121@tcp
lustre-MGS0000: 192.168.203.121@tcp
lustre-OST0001: 192.168.203.121@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-MGS0000: 192.168.203.121@tcp
lustre-OST0001: 192.168.203.121@tcp
lustre-MDT0000: 192.168.203.121@tcp
lustre-OST0000: 192.168.203.121@tcp
lustre-MDT0001: 192.168.203.121@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-MGS0000: 192.168.203.121@tcp
lustre-OST0001: 192.168.203.121@tcp
lustre-MDT0000: 192.168.203.121@tcp
lustre-OST0000: 192.168.203.121@tcp
lustre-MDT0001: 192.168.203.121@tcp
--- END OF LDEV OUTPUT ---
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
PASS 92 (11s)

== conf-sanity test 93: register mulitple MDT at the same time ========================================================== 09:28:17 (1786800497)

 SKIP: conf-sanity test_93 needs >= 3 MDTs
SKIP 93 (3s)

== conf-sanity test 94: ldev outputs correct labels for file system name query ========================================================== 09:28:20 (1786800500)
----- /tmp/ldev.conf -----
oleg321-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg321-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg321-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg321-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
oleg321-server	-	lustre-MDT0001	zfs:lustre-mdt2/mdt2
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg321-server	192.168.203.121@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-MDT0000
lustre-MDT0001
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
lustre-MDT0001
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
PASS 94 (7s)

== conf-sanity test 95: ldev should only allow one label filter ========================================================== 09:28:27 (1786800507)
----- /tmp/ldev.conf -----
oleg321-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg321-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg321-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg321-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
oleg321-server	-	lustre-MDT0001	zfs:lustre-mdt2/mdt2
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg321-server	192.168.203.121@tcp
--- END /tmp/nids ---
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
PASS 95 (10s)

== conf-sanity test 96: ldev returns hostname and backend fs correctly in command sub ========================================================== 09:28:37 (1786800517)
----- /tmp/ldev.conf -----
oleg321-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg321-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg321-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg321-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
oleg321-server	-	lustre-MDT0001	zfs:lustre-mdt2/mdt2
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg321-server	192.168.203.121@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
oleg321-server-zfs
oleg321-server-zfs
oleg321-server-zfs
oleg321-server-zfs
oleg321-server-zfs
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
oleg321-server-zfs
oleg321-server-zfs
oleg321-server-zfs
oleg321-server-zfs
oleg321-server-zfs
--- END OF EXPECTED OUTPUT ---
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
PASS 96 (7s)

== conf-sanity test 97: ldev returns correct ouput when querying based on role ========================================================== 09:28:44 (1786800524)
----- /tmp/ldev.conf -----
oleg321-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg321-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg321-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg321-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
oleg321-server	-	lustre-MDT0001	zfs:lustre-mdt2/mdt2
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg321-server	192.168.203.121@tcp
--- END /tmp/nids ---

MDT role
-- START OF LDEV OUTPUT --
lustre-MDT0000
lustre-MDT0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
lustre-MDT0001
--- END OF EXPECTED OUTPUT ---

OST role
-- START OF LDEV OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---

MGS role
-- START OF LDEV OUTPUT --
lustre-MGS0000
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MGS0000
--- END OF EXPECTED OUTPUT ---
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
PASS 97 (9s)

== conf-sanity test 98: Buffer-overflow check while parsing mount_opts ========================================================== 09:28:53 (1786800533)
start mds service on oleg321-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
setup single mount lustre success
arg[0] = mount.lustre
arg[1] = -o
arg[2] = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
arg[3] = -v
arg[4] = 192.168.203.121@tcp:/lustre
arg[5] = /mnt/lustre
source = 192.168.203.121@tcp:/lustre (192.168.203.121@tcp:/lustre), target = /mnt/lustre
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
PASS 98 (143s)

== conf-sanity test 99: Adding meta_bg option ============ 09:31:16 (1786800676)

 SKIP: conf-sanity test_99 ldiskfs only test
SKIP 99 (3s)

== conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 09:31:20 (1786800680)
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
oleg321-server: oleg321-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg321-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
setup single mount lustre success
lustre-MDT0000:
lustre-OST0000:
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
PASS 100 (205s)

== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:34:44 (1786800884)
start mds service on oleg321-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
 - open/close 206 (time 1786800983.06 total 10.02 last 20.57)
 - open/close 416 (time 1786800993.06 total 20.02 last 21.00)
 - open/close 577 (time 1786801003.07 total 30.03 last 16.09)
 - open/close 746 (time 1786801014.85 total 41.81 last 14.34)
 - open/close 874 (time 1786801026.89 total 53.85 last 10.64)
 - open/close 1070 (time 1786801036.90 total 63.85 last 19.58)
 - open/close 1237 (time 1786801047.43 total 74.39 last 15.85)
 - open/close 1453 (time 1786801057.43 total 84.39 last 21.59)
 - open/close 1657 (time 1786801067.83 total 94.78 last 19.63)
 - open/close 1896 (time 1786801078.46 total 105.42 last 22.47)
 - open/close 2118 (time 1786801088.76 total 115.72 last 21.55)
 - open/close 2328 (time 1786801099.09 total 126.05 last 20.32)
 - open/close 2630 (time 1786801109.57 total 136.53 last 28.82)
 - open/close 2882 (time 1786801121.46 total 148.42 last 21.20)
 - open/close 3093 (time 1786801131.46 total 158.42 last 21.10)
 - open/close 3351 (time 1786801141.47 total 168.43 last 25.77)
 - open/close 3535 (time 1786801151.51 total 178.46 last 18.33)
 - open/close 3716 (time 1786801162.07 total 189.03 last 17.13)
 - open/close 3933 (time 1786801172.73 total 199.68 last 20.37)
 - open/close 4221 (time 1786801183.69 total 210.65 last 26.27)
 - open/close 4411 (time 1786801193.69 total 220.65 last 18.99)
 - open/close 4573 (time 1786801204.16 total 231.11 last 15.49)
 - open/close 4733 (time 1786801214.18 total 241.14 last 15.96)
 - open/close 4938 (time 1786801224.77 total 251.72 last 19.36)
 - open/close 5178 (time 1786801234.77 total 261.73 last 23.99)
 - open/close 5407 (time 1786801245.36 total 272.32 last 21.62)
 - open/close 5662 (time 1786801255.37 total 282.33 last 25.48)
 - open/close 5880 (time 1786801267.28 total 294.23 last 18.31)
 - open/close 6103 (time 1786801277.30 total 304.25 last 22.25)
 - open/close 6322 (time 1786801287.33 total 314.28 last 21.84)
 - open/close 6585 (time 1786801297.91 total 324.87 last 24.85)
 - open/close 6830 (time 1786801308.27 total 335.23 last 23.64)
 - open/close 7110 (time 1786801319.03 total 345.99 last 26.04)
 - open/close 7391 (time 1786801329.87 total 356.82 last 25.93)
 - open/close 7680 (time 1786801340.36 total 367.31 last 27.55)
 - open/close 7919 (time 1786801350.37 total 377.33 last 23.87)
 - open/close 8157 (time 1786801361.08 total 388.04 last 22.21)
 - open/close 8376 (time 1786801371.09 total 398.05 last 21.88)
 - open/close 8563 (time 1786801381.82 total 408.77 last 17.44)
 - open/close 8710 (time 1786801391.82 total 418.78 last 14.69)
 - open/close 8872 (time 1786801402.00 total 428.95 last 15.92)
 - open/close 9029 (time 1786801412.29 total 439.25 last 15.25)
 - open/close 9182 (time 1786801422.90 total 449.86 last 14.41)
 - open/close 9361 (time 1786801433.37 total 460.33 last 17.09)
 - open/close 9586 (time 1786801443.77 total 470.73 last 21.64)
 - open/close 9837 (time 1786801453.77 total 480.73 last 25.10)
 - open/close 10000 (time 1786801460.65 total 487.61 last 23.70)
 - open/close 10285 (time 1786801471.17 total 498.13 last 27.10)
 - open/close 10439 (time 1786801481.76 total 508.72 last 14.54)
 - open/close 10575 (time 1786801492.74 total 519.70 last 12.39)
 - open/close 10846 (time 1786801503.69 total 530.65 last 24.74)
 - open/close 11098 (time 1786801514.10 total 541.06 last 24.21)
 - open/close 11314 (time 1786801524.12 total 551.07 last 21.57)
 - open/close 11559 (time 1786801534.80 total 561.76 last 22.93)
 - open/close 11759 (time 1786801544.81 total 571.76 last 19.99)
 - open/close 11962 (time 1786801554.83 total 581.78 last 20.26)
 - open/close 12093 (time 1786801564.84 total 591.80 last 13.08)
 - open/close 12246 (time 1786801575.61 total 602.57 last 14.20)
 - open/close 12514 (time 1786801586.68 total 613.64 last 24.22)
 - open/close 12690 (time 1786801597.38 total 624.34 last 16.44)
 - open/close 12857 (time 1786801607.39 total 634.35 last 16.69)
 - open/close 13095 (time 1786801617.60 total 644.55 last 23.32)
 - open/close 13321 (time 1786801627.64 total 654.60 last 22.49)
 - open/close 13554 (time 1786801637.94 total 664.90 last 22.63)
 - open/close 13814 (time 1786801647.94 total 674.90 last 26.00)
 - open/close 13985 (time 1786801659.06 total 686.01 last 15.39)
 - open/close 14204 (time 1786801669.99 total 696.95 last 20.03)
 - open/close 14455 (time 1786801680.12 total 707.07 last 24.79)
 - open/close 14722 (time 1786801690.12 total 717.08 last 26.70)
 - open/close 14960 (time 1786801700.13 total 727.09 last 23.77)
 - open/close 15193 (time 1786801710.73 total 737.68 last 21.99)
 - open/close 15422 (time 1786801720.74 total 747.69 last 22.87)
 - open/close 15769 (time 1786801730.74 total 757.70 last 34.68)
 - open/close 15882 (time 1786801741.14 total 768.10 last 10.87)
 - open/close 16138 (time 1786801751.60 total 778.56 last 24.47)
 - open/close 16403 (time 1786801761.65 total 788.61 last 26.35)
 - open/close 16594 (time 1786801771.96 total 798.92 last 18.53)
 - open/close 16858 (time 1786801781.98 total 808.94 last 26.35)
 - open/close 17091 (time 1786801792.46 total 819.42 last 22.23)
 - open/close 17303 (time 1786801802.55 total 829.51 last 21.00)
 - open/close 17485 (time 1786801813.24 total 840.20 last 17.03)
 - open/close 17635 (time 1786801825.21 total 852.17 last 12.54)
 - open/close 17823 (time 1786801835.21 total 862.17 last 18.80)
 - open/close 17927 (time 1786801846.03 total 872.99 last 9.61)
 - open/close 18177 (time 1786801856.05 total 883.00 last 24.96)
 - open/close 18283 (time 1786801866.18 total 893.13 last 10.46)
 - open/close 18442 (time 1786801876.99 total 903.95 last 14.70)
 - open/close 18704 (time 1786801887.01 total 913.97 last 26.14)
 - open/close 18804 (time 1786801897.32 total 924.28 last 9.70)
 - open/close 18990 (time 1786801908.17 total 935.12 last 17.15)
 - open/close 19191 (time 1786801918.22 total 945.17 last 20.00)
 - open/close 19930 (time 1786801928.22 total 955.17 last 73.89)
 - open/close 20506 (time 1786801938.22 total 965.18 last 57.58)
 - open/close 21358 (time 1786801948.23 total 975.18 last 85.16)
 - open/close 22097 (time 1786801958.23 total 985.18 last 73.90)
 - open/close 22659 (time 1786801968.23 total 995.19 last 56.16)
 - open/close 23291 (time 1786801978.24 total 1005.20 last 63.16)
 - open/close 23953 (time 1786801988.24 total 1015.20 last 66.17)
 - open/close 24788 (time 1786801998.26 total 1025.22 last 83.37)
 - open/close 25652 (time 1786802008.26 total 1035.22 last 86.39)
 - open/close 26500 (time 1786802018.26 total 1045.22 last 84.76)
 - open/close 27287 (time 1786802028.27 total 1055.23 last 78.64)
 - open/close 28011 (time 1786802038.28 total 1065.24 last 72.36)
 - open/close 28813 (time 1786802048.29 total 1075.24 last 80.13)
 - open/close 29604 (time 1786802058.32 total 1085.27 last 78.87)
 - open/close 30000 (time 1786802063.04 total 1090.00 last 83.77)
 - open/close 30720 (time 1786802073.05 total 1100.01 last 71.95)
 - open/close 31565 (time 1786802083.08 total 1110.04 last 84.24)
 - open/close 32273 (time 1786802093.09 total 1120.05 last 70.75)
 - open/close 33104 (time 1786802103.09 total 1130.05 last 83.06)
 - open/close 33830 (time 1786802113.10 total 1140.06 last 72.55)
 - open/close 34615 (time 1786802123.10 total 1150.06 last 78.49)
 - open/close 35448 (time 1786802133.10 total 1160.06 last 83.30)
 - open/close 36282 (time 1786802143.11 total 1170.07 last 83.33)
 - open/close 36976 (time 1786802153.12 total 1180.08 last 69.33)
 - open/close 37700 (time 1786802163.14 total 1190.09 last 72.28)
 - open/close 38481 (time 1786802173.14 total 1200.10 last 78.03)
 - open/close 39343 (time 1786802183.15 total 1210.11 last 86.12)
 - open/close 40000 (time 1786802191.24 total 1218.20 last 81.23)
 - open/close 40789 (time 1786802201.24 total 1228.20 last 78.88)
 - open/close 41787 (time 1786802211.26 total 1238.22 last 99.61)
 - open/close 42613 (time 1786802221.28 total 1248.24 last 82.43)
 - open/close 43220 (time 1786802231.30 total 1258.25 last 60.63)
 - open/close 43886 (time 1786802241.32 total 1268.28 last 66.46)
 - open/close 44651 (time 1786802251.32 total 1278.28 last 76.49)
 - open/close 45317 (time 1786802261.32 total 1288.28 last 66.59)
 - open/close 46079 (time 1786802271.33 total 1298.29 last 76.09)
 - open/close 46741 (time 1786802281.34 total 1308.30 last 66.17)
 - open/close 47329 (time 1786802291.35 total 1318.31 last 58.73)
 - open/close 47804 (time 1786802301.36 total 1328.32 last 47.45)
 - open/close 48528 (time 1786802311.37 total 1338.33 last 72.32)
 - open/close 49286 (time 1786802321.37 total 1348.33 last 75.80)
total: 50000 open/close in 1357.16 seconds: 36.84 ops/second
 - unlinked 0 (time 1786802334 ; total 0 ; last 0)
 - unlinked 10000 (time 1786802482 ; total 148 ; last 148)
 - unlinked 20000 (time 1786802620 ; total 286 ; last 138)
 - unlinked 30000 (time 1786802761 ; total 427 ; last 141)
 - unlinked 40000 (time 1786802901 ; total 567 ; last 140)
total: 50000 unlinks in 709 seconds: 70.521858 unlinks/second
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
PASS 101a (2199s)

== conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 10:11:23 (1786803083)
start mds service on oleg321-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
fail_loc=0x80002107
fail_val=20
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
oleg321-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96878310b800.ost_server_uuid 50
osc.lustre-OST0000-osc-ffff96878310b800.ost_server_uuid in FULL state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
PASS 101b (189s)

 SKIP: conf-sanity test_102 skipping excluded test 102

== conf-sanity test 103: rename filesystem name ========== 10:14:33 (1786803273)
Checking servers environments
Checking clients oleg321-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Starting client oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Started clients oleg321-client.virtnet: 
192.168.203.121@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff968785bad800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff968785bad800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 3s: want 'procname_uid' got 'procname_uid'
disable quota as required
oleg321-server: Pool lustre.pool1 created
oleg321-server: Pool lustre.lustre created
oleg321-server: OST lustre-OST0000_UUID added to pool lustre.lustre
Waiting 90s for 'lustre-OST0000_UUID'
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:)
Stopping client oleg321-client.virtnet /mnt/lustre opts:
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
rename lustre to mylustre
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-mdt1/mdt1
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

   Permanent disk data:
Device:     lustre-mdt1/mdt1
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:svname=mylustre-MDT0000
cmd: zfs set mountpoint=/tmp/mntxup5NF lustre-mdt1/mdt1 && zfs set canmount=on lustre-mdt1/mdt1 && zfs mount lustre-mdt1/mdt1
cmd: zfs umount lustre-mdt1/mdt1 && zfs set canmount=off lustre-mdt1/mdt1 && zfs set mountpoint=none lustre-mdt1/mdt1
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=mylustre
  lustre:svname=mylustre-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-mdt2/mdt2
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

   Permanent disk data:
Device:     lustre-mdt2/mdt2
Target:     mylustre-MDT0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:svname=mylustre-MDT0001
cmd: zfs set mountpoint=/tmp/mntv6NytV lustre-mdt2/mdt2 && zfs set canmount=on lustre-mdt2/mdt2 && zfs mount lustre-mdt2/mdt2
cmd: zfs umount lustre-mdt2/mdt2 && zfs set canmount=off lustre-mdt2/mdt2 && zfs set mountpoint=none lustre-mdt2/mdt2
Writing lustre-mdt2/mdt2 properties
  lustre:mgsnode=192.168.203.121@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=1
  lustre:index=1
  lustre:fsname=mylustre
  lustre:svname=mylustre-MDT0001
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=mylustre-OST0000
cmd: zfs set mountpoint=/tmp/mntPPD4Wh lustre-ost1/ost1 && zfs set canmount=on lustre-ost1/ost1 && zfs mount lustre-ost1/ost1
cmd: zfs umount lustre-ost1/ost1 && zfs set canmount=off lustre-ost1/ost1 && zfs set mountpoint=none lustre-ost1/ost1
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.121@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=mylustre
  lustre:svname=mylustre-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost2/ost2
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost2/ost2
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=mylustre-OST0001
cmd: zfs set mountpoint=/tmp/mnt0pAD1s lustre-ost2/ost2 && zfs set canmount=on lustre-ost2/ost2 && zfs mount lustre-ost2/ost2
cmd: zfs umount lustre-ost2/ost2 && zfs set canmount=off lustre-ost2/ost2 && zfs set mountpoint=none lustre-ost2/ost2
Writing lustre-ost2/ost2 properties
  lustre:mgsnode=192.168.203.121@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=mylustre
  lustre:svname=mylustre-OST0001
Checking servers environments
Checking clients oleg321-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  mylustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started mylustre-MDT0000
Start mds2: mount -t lustre -o localrecov  mylustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started mylustre-MDT0001
Start ost1: mount -t lustre -o localrecov  mylustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-mylustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started mylustre-OST0000
Start ost2: mount -t lustre -o localrecov  mylustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-mylustre-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started mylustre-OST0001
mount mylustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/mylustre /mnt/lustre
Starting client oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/mylustre /mnt/lustre
Started clients oleg321-client.virtnet: 
192.168.203.121@tcp:/mylustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.mylustre-OST0000-osc-ffff968785988000.idle_timeout=debug
osc.mylustre-OST0001-osc-ffff968785988000.idle_timeout=debug
disable quota as required
  File: /mnt/lustre/d103.conf-sanity/test-framework.sh
  Size: 315403    	Blocks: 1027       IO Block: 4194304 regular file
Device: c3aa56ceh/3282720462d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-08-15 10:16:24.000000000 -0400
Modify: 2026-08-15 10:16:24.000000000 -0400
Change: 2026-08-15 10:16:24.000000000 -0400
 Birth: 2026-08-15 10:16:23.000000000 -0400
Pool: mylustre.pool1
Pool: mylustre.lustre
mylustre-OST0000_UUID
mylustre-OST0000_UUID
oleg321-server: OST mylustre-OST0001_UUID added to pool mylustre.lustre
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:)
Stopping client oleg321-client.virtnet /mnt/lustre opts:
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
rename mylustre to tfs
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-mdt1/mdt1
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

   Permanent disk data:
Device:     mylustre-mdt1/mdt1
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:svname=tfs-MDT0000
cmd: zfs set mountpoint=/tmp/mnt1IyChE mylustre-mdt1/mdt1 && zfs set canmount=on mylustre-mdt1/mdt1 && zfs mount mylustre-mdt1/mdt1
cmd: zfs umount mylustre-mdt1/mdt1 && zfs set canmount=off mylustre-mdt1/mdt1 && zfs set mountpoint=none mylustre-mdt1/mdt1
Writing mylustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=tfs
  lustre:svname=tfs-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-mdt2/mdt2
Target:     mylustre-MDT0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

   Permanent disk data:
Device:     mylustre-mdt2/mdt2
Target:     tfs-MDT0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:svname=tfs-MDT0001
cmd: zfs set mountpoint=/tmp/mntMSy3lm mylustre-mdt2/mdt2 && zfs set canmount=on mylustre-mdt2/mdt2 && zfs mount mylustre-mdt2/mdt2
cmd: zfs umount mylustre-mdt2/mdt2 && zfs set canmount=off mylustre-mdt2/mdt2 && zfs set mountpoint=none mylustre-mdt2/mdt2
Writing mylustre-mdt2/mdt2 properties
  lustre:mgsnode=192.168.203.121@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=1
  lustre:index=1
  lustre:fsname=tfs
  lustre:svname=tfs-MDT0001
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-ost1/ost1
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     mylustre-ost1/ost1
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=tfs-OST0000
cmd: zfs set mountpoint=/tmp/mntVbFldw mylustre-ost1/ost1 && zfs set canmount=on mylustre-ost1/ost1 && zfs mount mylustre-ost1/ost1
cmd: zfs umount mylustre-ost1/ost1 && zfs set canmount=off mylustre-ost1/ost1 && zfs set mountpoint=none mylustre-ost1/ost1
Writing mylustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.121@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=tfs
  lustre:svname=tfs-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     mylustre-ost2/ost2
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     mylustre-ost2/ost2
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=tfs-OST0001
cmd: zfs set mountpoint=/tmp/mnteaCP0j mylustre-ost2/ost2 && zfs set canmount=on mylustre-ost2/ost2 && zfs mount mylustre-ost2/ost2
cmd: zfs umount mylustre-ost2/ost2 && zfs set canmount=off mylustre-ost2/ost2 && zfs set mountpoint=none mylustre-ost2/ost2
Writing mylustre-ost2/ost2 properties
  lustre:mgsnode=192.168.203.121@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=tfs
  lustre:svname=tfs-OST0001
Checking servers environments
Checking clients oleg321-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  tfs-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started tfs-MDT0000
Start mds2: mount -t lustre -o localrecov  tfs-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started tfs-MDT0001
Start ost1: mount -t lustre -o localrecov  tfs-ost1/ost1 /mnt/lustre-ost1
seq.cli-tfs-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started tfs-OST0000
Start ost2: mount -t lustre -o localrecov  tfs-ost2/ost2 /mnt/lustre-ost2
seq.cli-tfs-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started tfs-OST0001
mount tfs  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/tfs /mnt/lustre
Starting client oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/tfs /mnt/lustre
Started clients oleg321-client.virtnet: 
192.168.203.121@tcp:/tfs on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.tfs-OST0000-osc-ffff968785bab800.idle_timeout=debug
osc.tfs-OST0001-osc-ffff968785bab800.idle_timeout=debug
disable quota as required
  File: /mnt/lustre/d103.conf-sanity/test-framework.sh
  Size: 315403    	Blocks: 1027       IO Block: 4194304 regular file
Device: 32e2fa5ah/853736026d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-08-15 10:16:24.000000000 -0400
Modify: 2026-08-15 10:16:24.000000000 -0400
Change: 2026-08-15 10:16:24.000000000 -0400
 Birth: 2026-08-15 10:16:23.000000000 -0400
Pool: tfs.pool1
Pool: tfs.lustre
tfs-OST0000_UUID
tfs-OST0001_UUID
tfs-OST0000_UUID
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:)
Stopping client oleg321-client.virtnet /mnt/lustre opts:
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
rename tfs to lustre
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-mdt1/mdt1
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

   Permanent disk data:
Device:     tfs-mdt1/mdt1
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:svname=lustre-MDT0000
cmd: zfs set mountpoint=/tmp/mntPkVJbR tfs-mdt1/mdt1 && zfs set canmount=on tfs-mdt1/mdt1 && zfs mount tfs-mdt1/mdt1
cmd: zfs umount tfs-mdt1/mdt1 && zfs set canmount=off tfs-mdt1/mdt1 && zfs set mountpoint=none tfs-mdt1/mdt1
Writing tfs-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-mdt2/mdt2
Target:     tfs-MDT0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

   Permanent disk data:
Device:     tfs-mdt2/mdt2
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:svname=lustre-MDT0001
cmd: zfs set mountpoint=/tmp/mnt71EQhi tfs-mdt2/mdt2 && zfs set canmount=on tfs-mdt2/mdt2 && zfs mount tfs-mdt2/mdt2
cmd: zfs umount tfs-mdt2/mdt2 && zfs set canmount=off tfs-mdt2/mdt2 && zfs set mountpoint=none tfs-mdt2/mdt2
Writing tfs-mdt2/mdt2 properties
  lustre:mgsnode=192.168.203.121@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=1
  lustre:index=1
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0001
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-ost1/ost1
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     tfs-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
cmd: zfs set mountpoint=/tmp/mnttsn728 tfs-ost1/ost1 && zfs set canmount=on tfs-ost1/ost1 && zfs mount tfs-ost1/ost1
cmd: zfs umount tfs-ost1/ost1 && zfs set canmount=off tfs-ost1/ost1 && zfs set mountpoint=none tfs-ost1/ost1
Writing tfs-ost1/ost1 properties
  lustre:mgsnode=192.168.203.121@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     tfs-ost2/ost2
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     tfs-ost2/ost2
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.121@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0001
cmd: zfs set mountpoint=/tmp/mntgE17b0 tfs-ost2/ost2 && zfs set canmount=on tfs-ost2/ost2 && zfs mount tfs-ost2/ost2
cmd: zfs umount tfs-ost2/ost2 && zfs set canmount=off tfs-ost2/ost2 && zfs set mountpoint=none tfs-ost2/ost2
Writing tfs-ost2/ost2 properties
  lustre:mgsnode=192.168.203.121@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=lustre
  lustre:svname=lustre-OST0001
Checking servers environments
Checking clients oleg321-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Starting client oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
Started clients oleg321-client.virtnet: 
192.168.203.121@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff968787af3800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff968787af3800.idle_timeout=debug
disable quota as required
PASS 103 (492s)

== conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 10:22:45 (1786803765)
mountfsopt: acl,user_xattr
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg321-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
oleg321-server: oleg321-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
Start mds1: mount -t lustre -o localrecov,noacl  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov,noacl  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
setfacl: /mnt/lustre: Operation not supported
PASS 104a (184s)

== conf-sanity test 104b: Mount uses last flock argument ========================================================== 10:25:50 (1786803950)
mount lustre with opts flock,localflock on /mnt/lustre3.....
Starting client: oleg321-client.virtnet:  -o flock,localflock 192.168.203.121@tcp:/lustre /mnt/lustre3
192.168.203.121@tcp:/lustre on /mnt/lustre3 type lustre (rw,localflock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose)
umount lustre on /mnt/lustre3.....
Stopping client oleg321-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock on /mnt/lustre3.....
Starting client: oleg321-client.virtnet:  -o localflock,flock 192.168.203.121@tcp:/lustre /mnt/lustre3
192.168.203.121@tcp:/lustre on /mnt/lustre3 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose)
umount lustre on /mnt/lustre3.....
Stopping client oleg321-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock,noflock on /mnt/lustre3.....
Starting client: oleg321-client.virtnet:  -o localflock,flock,noflock 192.168.203.121@tcp:/lustre /mnt/lustre3
umount lustre on /mnt/lustre3.....
Stopping client oleg321-client.virtnet /mnt/lustre3 (opts:)
PASS 104b (14s)

== conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 10:26:04 (1786803964)
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
oleg321-server: rmmod: ERROR: Module mdd is in use
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg321-server
oleg321-server: oleg321-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg321-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg321-client.virtnet:  -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre
touch: cannot touch '/tmp/d105.conf-sanity/f105.conf-sanity': Read-only file system
umount lustre on /mnt/lustre.....
Stopping client oleg321-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
PASS 105 (225s)

 SKIP: conf-sanity test_106 skipping SLOW test 106

== conf-sanity test 107: Unknown config param should not fail target mounting ========================================================== 10:29:52 (1786804192)
start mds service on oleg321-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg321-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
start ost1 service on oleg321-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1
Started lustre-OST0000
oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
umount lustre on /mnt/lustre.....
stop ost1 service on oleg321-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server
stop mds service on oleg321-server
stop mds service on oleg321-server

unloading modules via unload_modules_local on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing unload_modules_local
oleg321-server: 
modules unloaded.
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2
PASS 107 (242s)

 SKIP: conf-sanity test_108a skipping ALWAYS excluded test 108a

== conf-sanity test 108b: migrate from ZFS to ldiskfs ==== 10:33:55 (1786804435)

 SKIP: conf-sanity test_108b ldiskfs only test
SKIP 108b (3s)

== conf-sanity test 109a: test lctl clear_conf fsname ==== 10:33:58 (1786804438)

 SKIP: conf-sanity test_109a LU-8727: no implementation for ZFS
SKIP 109a (4s)

== conf-sanity test 109b: test lctl clear_conf one config ========================================================== 10:34:02 (1786804442)

 SKIP: conf-sanity test_109b LU-8727: no implementation for ZFS
SKIP 109b (3s)

== conf-sanity test 110: Adding large_dir with 3-level htree ========================================================== 10:34:05 (1786804445)

 SKIP: conf-sanity test_110 Only applicable to ldiskfs-based MDTs
SKIP 110 (3s)

 SKIP: conf-sanity test_111 skipping SLOW test 111

 SKIP: conf-sanity test_112a skipping ALWAYS excluded test 112a

 SKIP: conf-sanity test_112b skipping ALWAYS excluded test 112b

 SKIP: conf-sanity test_113 skipping ALWAYS excluded test 113

 SKIP: conf-sanity test_114 skipping SLOW test 114

 SKIP: conf-sanity test_115 skipping excluded test 115

== conf-sanity test 116: big size MDT support ============ 10:34:19 (1786804459)

 SKIP: conf-sanity test_116 ldiskfs only test
SKIP 116 (3s)

 SKIP: conf-sanity test_117 skipping ALWAYS excluded test 117

 SKIP: conf-sanity test_119 skipping ALWAYS excluded test 119

== conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 10:34:25 (1786804465)

 SKIP: conf-sanity test_120 ldiskfs only test
SKIP 120 (4s)
Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2
oleg321-server: oleg321-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg321-server'
oleg321-server: oleg321-server.virtnet: executing load_modules_local
oleg321-server: Loading modules from /home/green/git/lustre-release/lustre
oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg321-server: detected 4 online CPUs by sysfs
oleg321-server: MODOPTS_LIBCFS=
oleg321-server: Force libcfs to create 2 CPU partitions
oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
== conf-sanity test complete, duration 7102 sec ========== 10:35:47 (1786804547)
=== conf-sanity: start cleanup 10:35:49 (1786804549) ===
=== conf-sanity: finish cleanup 10:35:52 (1786804552) ===