-----============= acceptance-small: conf-sanity ============----- Thu Jan 22 04:28:32 EST 2026
mgs: Rocky Linux release 8.10 (Green Obsidian)
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
mds1: Rocky Linux release 8.10 (Green Obsidian)
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
ost1: Rocky Linux release 8.10 (Green Obsidian)
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
client: Rocky Linux release 8.10 (Green Obsidian)
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg241-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory
excepting tests: 102 106 115 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a
skipping tests SLOW=no: 45 69 106 111 114
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg241-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
oleg241-server: oleg241-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg241-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server

== conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 04:31:43 (1769074303)
start mds service on oleg241-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Change MGS params
max_dirty_mb: 462
new_max_dirty_mb: 452
Waiting 90s for '452'
Updated after 2s: want '452' got '452'
452
Check the value is stored after remount
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:)
Stopping client oleg241-client.virtnet /mnt/lustre opts:
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Checking servers environments
Checking clients oleg241-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Starting client oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Started clients oleg241-client.virtnet: 
192.168.202.141@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff99b898bd4800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff99b898bd4800.idle_timeout=debug
disable quota as required
Change OST params
client_cache_count: 128
new_client_cache_count: 256
Waiting 90s for '256'
Updated after 2s: want '256' got '256'
256
Check the value is stored after remount
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:)
Stopping client oleg241-client.virtnet /mnt/lustre opts:
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
Checking servers environments
Checking clients oleg241-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Starting client oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Started clients oleg241-client.virtnet: 
192.168.202.141@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff99b88925d800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff99b88925d800.idle_timeout=debug
disable quota as required
256
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:)
Stopping client oleg241-client.virtnet /mnt/lustre opts:
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
PASS 76a (337s)

== conf-sanity test 76b: verify params log setup correctly ========================================================== 04:37:19 (1769074639)
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:)
Checking servers environments
Checking clients oleg241-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Starting client oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Started clients oleg241-client.virtnet: 
192.168.202.141@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff99b888f6a800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff99b888f6a800.idle_timeout=debug
disable quota as required
mgs.MGS.live.params=
fsname: params
flags: 0x20     gen: 1

Secure RPC Config Rules:

imperative_recovery_state:
    state: startup
    nonir_clients: 0
    nidtbl_version: 2
    notify_duration_total: 0.000000000
    notify_duation_max: 0.000000000
    notify_count: 0
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:)
Stopping client oleg241-client.virtnet /mnt/lustre opts:
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
PASS 76b (154s)

== conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 04:39:53 (1769074793)
Checking servers environments
Checking clients oleg241-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Starting client oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Started clients oleg241-client.virtnet: 
192.168.202.141@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff99b888e99800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff99b888e99800.idle_timeout=debug
disable quota as required
Change changelog_mask
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Check the value is stored after mds remount
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
start mds service on oleg241-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:)
Stopping client oleg241-client.virtnet /mnt/lustre opts:
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
PASS 76c (166s)

== conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 04:42:39 (1769074959)
Checking servers environments
Checking clients oleg241-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Starting client oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Started clients oleg241-client.virtnet: 
192.168.202.141@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff99b8893c8800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff99b8893c8800.idle_timeout=debug
disable quota as required
lctl set_param -P llite.*.xattr_cache=0
Waiting 90s for '0'
Updated after 2s: want '0' got '0'
Check llite.$inst.xattr_cache on client /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Check llite.$inst.xattr_cache on the new client /mnt/lustre2
mount lustre  on /mnt/lustre2.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre2
umount lustre on /mnt/lustre2.....
Stopping client oleg241-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:)
Stopping client oleg241-client.virtnet /mnt/lustre opts:
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
oleg241-server: No device found for name MGS: Invalid argument
oleg241-server: This command must be run on the MGS.
oleg241-server: error: executing set_param: No such device
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 19
Reader error: 'No devices found' at 0
PASS 76d (135s)

== conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 04:44:55 (1769075095)
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1

   Permanent disk data:
Target:     test1234:MDT0000
Index:      0
Lustre FS:  test1234
Mount type: zfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1_2/mdt1_2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1_2/mdt1_2 properties
  lustre:mgsnode=192.168.202.141@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=101
  lustre:index=0
  lustre:fsname=test1234
  lustre:svname=test1234:MDT0000
Start fs2mds: mount -t lustre -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1_2/mdt1_2
Started test1234-MDT0000

   Permanent disk data:
Target:     test1234:OST0000
Index:      0
Lustre FS:  test1234
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=0.0.0.0@tcp,192.168.202.141@tcp,192.168.202.141@tcp:192.168.202.141@tcp:192.168.202.141@tcp sys.timeout=20 failover.node=1.2.3.4@tcp,4.3.2.1@tcp autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1_2/ost1_2
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1_2/ost1_2 properties
  lustre:mgsnode=0.0.0.0@tcp,192.168.202.141@tcp,192.168.202.141@tcp:192.168.202.141@tcp:192.168.202.141@tcp
  lustre:sys.timeout=20
  lustre:failover.node=1.2.3.4@tcp,4.3.2.1@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=test1234
  lustre:svname=test1234:OST0000
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1777: OST_INDICES: bad array subscript
Start fs2ost: mount -t lustre -o localrecov  lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost
seq.cli-test1234-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-ost1_2/ost1_2
Started test1234-OST0000
oleg241-server: fstrim: /mnt/lustre-fs2ost: the discard operation is not supported
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
setup single mount lustre success
umount /mnt/lustre2 ...
stopping fs2mds ...
Stopping /mnt/lustre-fs2mds (opts:-f) on oleg241-server
stopping fs2ost ...
Stopping /mnt/lustre-fs2ost (opts:-f) on oleg241-server
PASS 77 (85s)

== conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 04:46:20 (1769075180)

 SKIP: conf-sanity test_78 ldiskfs only test
SKIP 78 (4s)

== conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 04:46:24 (1769075184)
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: 
oleg241-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg241-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 22
oleg241-server: 
oleg241-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg241-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 22
oleg241-server: 
oleg241-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg241-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 22
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f)
oleg241-server: oleg241-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg241-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
PASS 79 (201s)

== conf-sanity test 80: mgc import reconnect race ======== 04:49:45 (1769075385)
start mds service on oleg241-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
fail_val=10
fail_loc=0x906
fail_val=10
fail_loc=0x906
start ost2 service on oleg241-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
fail_loc=0
stop ost2 service on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
PASS 80 (144s)

== conf-sanity test 81: sparse OST indexing ============== 04:52:09 (1769075529)

 SKIP: conf-sanity test_81 needs >= 3 OSTs
SKIP 81 (4s)

== conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 04:52:13 (1769075533)

 SKIP: conf-sanity test_82a needs >= 3 OSTs
SKIP 82a (4s)

== conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 04:52:17 (1769075537)

 SKIP: conf-sanity test_82b needs >= 4 OSTs
SKIP 82b (3s)

== conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 04:52:21 (1769075541)

 SKIP: conf-sanity test_82c needs >= 4 OSTs
SKIP 82c (4s)

== conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 04:52:25 (1769075545)

 SKIP: conf-sanity test_83 ldiskfs only test
SKIP 83 (4s)

== conf-sanity test 84: check recovery_hard_time ========= 04:52:29 (1769075549)
start mds service on oleg241-server
start mds service on oleg241-server
Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
start ost2 service on oleg241-server
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0001
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg241-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
recovery_time=60, timeout=20, wrap_up=5
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre2.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre2
UUID                   1K-blocks        Used   Available Use%  Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1%  /mnt/lustre[MDT:0]
lustre-OST0000_UUID       382976        3072      377856   1%  /mnt/lustre[OST:0]
lustre-OST0001_UUID       382976        3072      377856   1%  /mnt/lustre[OST:1]

filesystem_summary:       765952        6144      755712   1%  /mnt/lustre

 - open/close 339 (time 1769075660.58 total 10.01 last 33.85)
 - open/close 719 (time 1769075670.60 total 20.03 last 37.93)
total: 1000 open/close in 26.23 seconds: 38.12 ops/second
fail_loc=0x20000709
fail_val=5
Failing mds1 on oleg241-server
Stopping /mnt/lustre-mds1 (opts:) on oleg241-server
04:54:45 (1769075685) shut down
facet: mds1 facet_host: oleg241-server facet_failover_host: oleg241-server
Failover mds1 to oleg241-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
04:55:20 (1769075720) targets are mounted
04:55:20 (1769075720) facet_failover done
oleg241-client: error: invalid path '/mnt/lustre': Input/output error
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 5
recovery status
status: COMPLETE
recovery_start: 1769075713
recovery_duration: 60
completed_clients: 1/2
replayed_requests: 188
last_transno: 12884902080
VBR: DISABLED
IR: DISABLED
fail_loc=0
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
umount lustre on /mnt/lustre2.....
Stopping client oleg241-client.virtnet /mnt/lustre2 (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop ost2 service on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
PASS 84 (282s)

== conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 04:57:11 (1769075831)
fail_loc=0x197
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
PASS 85 (85s)

== conf-sanity test 86: Replacing mkfs.lustre -G option == 04:58:36 (1769075916)

 SKIP: conf-sanity test_86 LU-6442: no such mkfs params for ZFS OSTs
SKIP 86 (4s)

== conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 04:58:40 (1769075920)

 SKIP: conf-sanity test_87 ldiskfs only test
SKIP 87 (3s)

== conf-sanity test 88: check the default mount options can be overridden ========================================================== 04:58:44 (1769075924)

 SKIP: conf-sanity test_88 LU-6662: no implementation for ZFS
SKIP 88 (4s)

== conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 04:58:48 (1769075928)
Failing mds1 on oleg241-server
04:58:54 (1769075934) shut down
facet: mds1 facet_host: oleg241-server facet_failover_host: oleg241-server
Failover mds1 to oleg241-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
04:59:19 (1769075959) targets are mounted
04:59:19 (1769075959) facet_failover done
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 95
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 95
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
tunefs --param failover.node=192.0.2.254@tcp0
tunefs --param failover.node=192.0.2.255@tcp0
tunefs --erase-param failover.node
tunefs --erase-params
tunefs --param failover.node=192.0.2.254@tcp0 --erase-params
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f)
oleg241-server: oleg241-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg241-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
PASS 89 (250s)

== conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 05:02:58 (1769076178)
start mds service on oleg241-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 8 files ...
fail_loc=0x159
launch 6 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-7 has perms 0600 OK
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-8 has perms 0644 OK
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 90a (125s)

== conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ========================================================== 05:05:03 (1769076303)
start mds service on oleg241-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff99b8852e8000.max_mod_rpcs_in_flight=1
max_mod_rpcs_in_flight set to 1
creating 2 files ...
fail_loc=0x159
launch 0 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-1 has perms 0600 OK
fail_loc=0x159
launch 1 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-2 has perms 0644 OK
mdc.lustre-MDT0000-mdc-ffff99b8852e8000.max_mod_rpcs_in_flight=5
max_mod_rpcs_in_flight set to 5
creating 6 files ...
fail_loc=0x159
launch 4 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-5 has perms 0600 OK
fail_loc=0x159
launch 5 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-6 has perms 0644 OK
mdt_max_mod_rpcs_in_flight is 8
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff99b8915d3800.max_rpcs_in_flight=17
mdc.lustre-MDT0000-mdc-ffff99b8915d3800.max_mod_rpcs_in_flight=16
max_mod_rpcs_in_flight set to 16
creating 17 files ...
fail_loc=0x159
launch 15 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-16 has perms 0600 OK
fail_loc=0x159
launch 16 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-17 has perms 0644 OK
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 90b (237s)

== conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ========================================================== 05:09:00 (1769076540)
start mds service on oleg241-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
max_rpcs_in_flight is 8
MDC max_mod_rpcs_in_flight is 7
mdt_max_mod_rpcs_in_flight is 8
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mdc.lustre-MDT0000-mdc-ffff99b888580800.max_mod_rpcs_in_flight=8
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff99b8852e9000.max_rpcs_in_flight=10
error: set_param: setting /sys/fs/lustre/mdc/lustre-MDT0000-mdc-ffff99b8852e9000/max_mod_rpcs_in_flight=9: Numerical result out of range
error: set_param: setting 'mdc/lustre-MDT0000-mdc-*/max_mod_rpcs_in_flight'='9': Numerical result out of range
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
PASS 90c (143s)

== conf-sanity test 90d: check one close RPC is allowed above max_mod_rpcs_in_flight ========================================================== 05:11:23 (1769076683)
start mds service on oleg241-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 7 files ...
multiop /mnt/lustre/d90d.conf-sanity/file-close vO_c
TMPPIPE=/tmp/multiop_open_wait_pipe.7581
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional close in parallel ...
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 90d (156s)

== conf-sanity test 91: evict-by-nid support ============= 05:13:59 (1769076839)
start mds service on oleg241-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
setup single mount lustre success
list nids on mdt:
mdt.lustre-MDT0000.exports.0@lo
mdt.lustre-MDT0000.exports.192.168.202.41@tcp
mdt.lustre-MDT0000.exports.clear
uuid from 192\.168\.202\.41@tcp:
mdt.lustre-MDT0000.exports.192.168.202.41@tcp.uuid=6e0f6ad9-3dd7-46c2-9934-b44d96b8d828
manual umount lustre on /mnt/lustre....
evict 192\.168\.202\.41@tcp
oleg241-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.202.41@tcp/uuid': No such device
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 19
oleg241-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.202.41@tcp/uuid': No such device
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 19
oleg241-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.202.41@tcp/uuid': No such device
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 19
oleg241-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.202.41@tcp/uuid': No such device
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 19
umount lustre on /mnt/lustre.....
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
PASS 91 (176s)

== conf-sanity test 92: ldev returns MGS NID correctly in command substitution ========================================================== 05:16:55 (1769077015)
Host is oleg241-client.virtnet
----- /tmp/ldev.conf -----
oleg241-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg241-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg241-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg241-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg241-server	192.168.202.141@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-MGS0000: 192.168.202.141@tcp
lustre-MDT0000: 192.168.202.141@tcp
lustre-OST0001: 192.168.202.141@tcp
lustre-OST0000: 192.168.202.141@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0001: 192.168.202.141@tcp
lustre-OST0000: 192.168.202.141@tcp
lustre-MGS0000: 192.168.202.141@tcp
lustre-MDT0000: 192.168.202.141@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.202.141@tcp
lustre-MGS0000: 192.168.202.141@tcp
lustre-OST0001: 192.168.202.141@tcp
lustre-MDT0000: 192.168.202.141@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0001: 192.168.202.141@tcp
lustre-MGS0000: 192.168.202.141@tcp
lustre-OST0000: 192.168.202.141@tcp
lustre-MDT0000: 192.168.202.141@tcp
--- END OF LDEV OUTPUT ---
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 92 (10s)

== conf-sanity test 93: register mulitple MDT at the same time ========================================================== 05:17:06 (1769077026)

 SKIP: conf-sanity test_93 needs >= 3 MDTs
SKIP 93 (3s)

== conf-sanity test 94: ldev outputs correct labels for file system name query ========================================================== 05:17:09 (1769077029)
----- /tmp/ldev.conf -----
oleg241-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg241-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg241-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg241-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg241-server	192.168.202.141@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-MDT0000
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 94 (9s)

== conf-sanity test 95: ldev should only allow one label filter ========================================================== 05:17:18 (1769077038)
----- /tmp/ldev.conf -----
oleg241-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg241-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg241-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg241-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg241-server	192.168.202.141@tcp
--- END /tmp/nids ---
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 95 (10s)

== conf-sanity test 96: ldev returns hostname and backend fs correctly in command sub ========================================================== 05:17:28 (1769077048)
----- /tmp/ldev.conf -----
oleg241-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg241-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg241-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg241-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg241-server	192.168.202.141@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
oleg241-server-zfs
oleg241-server-zfs
oleg241-server-zfs
oleg241-server-zfs
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
oleg241-server-zfs
oleg241-server-zfs
oleg241-server-zfs
oleg241-server-zfs
--- END OF EXPECTED OUTPUT ---
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 96 (9s)

== conf-sanity test 97: ldev returns correct ouput when querying based on role ========================================================== 05:17:37 (1769077057)
----- /tmp/ldev.conf -----
oleg241-server	-	lustre-MGS0000	zfs:lustre-mdt1/mdt1
oleg241-server	-	lustre-OST0000	zfs:lustre-ost1/ost1
oleg241-server	-	lustre-OST0001	zfs:lustre-ost2/ost2
oleg241-server	-	lustre-MDT0000	zfs:lustre-mdt1/mdt1
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg241-server	192.168.202.141@tcp
--- END /tmp/nids ---

MDT role
-- START OF LDEV OUTPUT --
lustre-MDT0000
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
--- END OF EXPECTED OUTPUT ---

OST role
-- START OF LDEV OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---

MGS role
-- START OF LDEV OUTPUT --
lustre-MGS0000
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MGS0000
--- END OF EXPECTED OUTPUT ---
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 97 (9s)

== conf-sanity test 98: Buffer-overflow check while parsing mount_opts ========================================================== 05:17:46 (1769077066)
start mds service on oleg241-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
setup single mount lustre success
arg[0] = mount.lustre
arg[1] = -o
arg[2] = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
arg[3] = -v
arg[4] = 192.168.202.141@tcp:/lustre
arg[5] = /mnt/lustre
source = 192.168.202.141@tcp:/lustre (192.168.202.141@tcp:/lustre), target = /mnt/lustre
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 98 (124s)

== conf-sanity test 99: Adding meta_bg option ============ 05:19:51 (1769077191)

 SKIP: conf-sanity test_99 ldiskfs only test
SKIP 99 (5s)

== conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:19:55 (1769077195)
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
oleg241-server: oleg241-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg241-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
setup single mount lustre success
lustre-MDT0000:
lustre-OST0000:
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 100 (185s)

== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:23:00 (1769077380)
start mds service on oleg241-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
 - open/close 100 (time 1769077473.53 total 10.26 last 9.75)
 - open/close 199 (time 1769077485.41 total 22.14 last 8.33)
 - open/close 400 (time 1769077495.43 total 32.17 last 20.04)
 - open/close 583 (time 1769077505.44 total 42.17 last 18.29)
 - open/close 782 (time 1769077515.88 total 52.62 last 19.05)
 - open/close 946 (time 1769077527.26 total 63.99 last 14.42)
 - open/close 1181 (time 1769077537.27 total 74.00 last 23.49)
 - open/close 1289 (time 1769077548.33 total 85.06 last 9.76)
 - open/close 1393 (time 1769077558.35 total 95.08 last 10.38)
 - open/close 1500 (time 1769077569.19 total 105.92 last 9.87)
 - open/close 1676 (time 1769077579.76 total 116.49 last 16.65)
 - open/close 1799 (time 1769077590.29 total 127.03 last 11.67)
 - open/close 2032 (time 1769077600.41 total 137.14 last 23.04)
 - open/close 2316 (time 1769077610.41 total 147.14 last 28.39)
 - open/close 2510 (time 1769077620.44 total 157.17 last 19.35)
 - open/close 2726 (time 1769077633.43 total 170.16 last 16.63)
 - open/close 2934 (time 1769077644.32 total 181.05 last 19.10)
 - open/close 3105 (time 1769077654.65 total 191.38 last 16.56)
 - open/close 3322 (time 1769077664.98 total 201.71 last 20.99)
 - open/close 3499 (time 1769077674.98 total 211.72 last 17.70)
 - open/close 3710 (time 1769077685.46 total 222.20 last 20.14)
 - open/close 3951 (time 1769077695.49 total 232.23 last 24.03)
 - open/close 4152 (time 1769077705.50 total 242.23 last 20.09)
 - open/close 4258 (time 1769077716.25 total 252.98 last 9.86)
 - open/close 4473 (time 1769077726.26 total 262.99 last 21.49)
 - open/close 4600 (time 1769077738.64 total 275.37 last 10.25)
 - open/close 4718 (time 1769077749.21 total 285.94 last 11.16)
 - open/close 4934 (time 1769077759.90 total 296.63 last 20.21)
 - open/close 5182 (time 1769077770.92 total 307.65 last 22.51)
 - open/close 5383 (time 1769077781.53 total 318.26 last 18.94)
 - open/close 5589 (time 1769077791.96 total 328.69 last 19.75)
 - open/close 5770 (time 1769077802.70 total 339.43 last 16.86)
 - open/close 6059 (time 1769077813.65 total 350.39 last 26.37)
 - open/close 6309 (time 1769077823.66 total 360.39 last 24.98)
 - open/close 6529 (time 1769077834.21 total 370.95 last 20.85)
 - open/close 6772 (time 1769077844.23 total 380.97 last 24.25)
 - open/close 6966 (time 1769077854.37 total 391.10 last 19.14)
 - open/close 7110 (time 1769077864.43 total 401.16 last 14.32)
 - open/close 7258 (time 1769077874.92 total 411.65 last 14.11)
 - open/close 7379 (time 1769077884.92 total 421.65 last 12.09)
 - open/close 7501 (time 1769077894.96 total 431.69 last 12.15)
 - open/close 7598 (time 1769077905.13 total 441.87 last 9.53)
 - open/close 7711 (time 1769077916.02 total 452.75 last 10.38)
 - open/close 7839 (time 1769077930.06 total 466.79 last 9.12)
 - open/close 7963 (time 1769077942.56 total 479.29 last 9.92)
 - open/close 8045 (time 1769077952.82 total 489.55 last 7.99)
 - open/close 8207 (time 1769077962.91 total 499.64 last 16.05)
 - open/close 8323 (time 1769077973.38 total 510.11 last 11.08)
 - open/close 8471 (time 1769077983.57 total 520.30 last 14.53)
 - open/close 8658 (time 1769077996.59 total 533.32 last 14.37)
 - open/close 8797 (time 1769078007.52 total 544.25 last 12.71)
 - open/close 9233 (time 1769078017.56 total 554.29 last 43.45)
 - open/close 9408 (time 1769078027.56 total 564.29 last 17.50)
 - open/close 9611 (time 1769078040.21 total 576.94 last 16.05)
 - open/close 9778 (time 1769078050.33 total 587.06 last 16.50)
 - open/close 9894 (time 1769078060.35 total 597.08 last 11.58)
 - open/close 10000 (time 1769078070.14 total 606.87 last 10.83)
 - open/close 10168 (time 1769078080.14 total 616.87 last 16.79)
 - open/close 10394 (time 1769078094.38 total 631.11 last 15.87)
 - open/close 10486 (time 1769078104.59 total 641.32 last 9.01)
 - open/close 10652 (time 1769078114.61 total 651.34 last 16.57)
 - open/close 10779 (time 1769078127.21 total 663.94 last 10.08)
 - open/close 10976 (time 1769078137.22 total 673.95 last 19.69)
 - open/close 11106 (time 1769078147.23 total 683.96 last 12.98)
 - open/close 11269 (time 1769078157.25 total 693.98 last 16.26)
 - open/close 11459 (time 1769078167.38 total 704.11 last 18.76)
 - open/close 11602 (time 1769078177.89 total 714.63 last 13.60)
 - open/close 11749 (time 1769078187.90 total 724.64 last 14.68)
 - open/close 11918 (time 1769078197.91 total 734.64 last 16.89)
 - open/close 12014 (time 1769078208.47 total 745.20 last 9.09)
 - open/close 12181 (time 1769078218.47 total 755.20 last 16.70)
 - open/close 12254 (time 1769078229.80 total 766.53 last 6.45)
 - open/close 12340 (time 1769078240.43 total 777.17 last 8.08)
 - open/close 12378 (time 1769078250.89 total 787.62 last 3.63)
 - open/close 12464 (time 1769078261.66 total 798.39 last 7.98)
 - open/close 12521 (time 1769078271.67 total 808.40 last 5.70)
 - open/close 12712 (time 1769078281.69 total 818.42 last 19.06)
 - open/close 12869 (time 1769078292.72 total 829.46 last 14.22)
 - open/close 12933 (time 1769078303.36 total 840.09 last 6.02)
 - open/close 13095 (time 1769078313.83 total 850.56 last 15.47)
 - open/close 13228 (time 1769078323.88 total 860.61 last 13.24)
 - open/close 13295 (time 1769078334.08 total 870.81 last 6.57)
 - open/close 13482 (time 1769078344.09 total 880.83 last 18.68)
 - open/close 13651 (time 1769078354.13 total 890.86 last 16.84)
 - open/close 13813 (time 1769078364.95 total 901.69 last 14.97)
 - open/close 14019 (time 1769078375.35 total 912.08 last 19.82)
 - open/close 14249 (time 1769078385.62 total 922.35 last 22.40)
 - open/close 14576 (time 1769078396.15 total 932.89 last 31.03)
 - open/close 14703 (time 1769078406.77 total 943.50 last 11.97)
 - open/close 14903 (time 1769078416.81 total 953.54 last 19.91)
 - open/close 15057 (time 1769078427.08 total 963.82 last 14.99)
 - open/close 15317 (time 1769078437.12 total 973.85 last 25.92)
 - open/close 15410 (time 1769078447.13 total 983.86 last 9.29)
 - open/close 15537 (time 1769078457.31 total 994.04 last 12.48)
 - open/close 15657 (time 1769078467.57 total 1004.31 last 11.69)
 - open/close 15931 (time 1769078477.58 total 1014.31 last 27.39)
 - open/close 15999 (time 1769078487.58 total 1024.31 last 6.80)
 - open/close 16054 (time 1769078498.63 total 1035.36 last 4.98)
 - open/close 16145 (time 1769078509.54 total 1046.28 last 8.34)
 - open/close 16319 (time 1769078520.05 total 1056.79 last 16.55)
 - open/close 16523 (time 1769078531.82 total 1068.56 last 17.33)
 - open/close 16782 (time 1769078542.74 total 1079.47 last 23.73)
 - open/close 16890 (time 1769078552.77 total 1089.50 last 10.76)
 - open/close 17134 (time 1769078562.77 total 1099.50 last 24.40)
 - open/close 17422 (time 1769078573.42 total 1110.15 last 27.04)
 - open/close 17619 (time 1769078583.43 total 1120.16 last 19.69)
 - open/close 18063 (time 1769078593.45 total 1130.19 last 44.27)
 - open/close 18542 (time 1769078603.46 total 1140.19 last 47.88)
 - open/close 19051 (time 1769078613.47 total 1150.20 last 50.85)
 - open/close 19678 (time 1769078623.47 total 1160.20 last 62.68)
 - open/close 20000 (time 1769078629.28 total 1166.01 last 55.45)
 - open/close 20591 (time 1769078639.28 total 1176.01 last 59.07)
 - open/close 21280 (time 1769078649.30 total 1186.04 last 68.74)
 - open/close 21885 (time 1769078659.31 total 1196.04 last 60.49)
 - open/close 22471 (time 1769078669.32 total 1206.05 last 58.53)
 - open/close 22985 (time 1769078679.33 total 1216.06 last 51.35)
 - open/close 23788 (time 1769078689.34 total 1226.07 last 80.25)
 - open/close 24500 (time 1769078699.35 total 1236.08 last 71.07)
 - open/close 25060 (time 1769078709.38 total 1246.11 last 55.86)
 - open/close 25643 (time 1769078719.38 total 1256.11 last 58.29)
 - open/close 26260 (time 1769078729.39 total 1266.12 last 61.65)
 - open/close 26811 (time 1769078739.41 total 1276.14 last 54.97)
 - open/close 27448 (time 1769078749.43 total 1286.16 last 63.58)
 - open/close 28144 (time 1769078759.43 total 1296.16 last 69.59)
 - open/close 28859 (time 1769078769.44 total 1306.17 last 71.47)
 - open/close 29543 (time 1769078779.44 total 1316.18 last 68.34)
 - open/close 30000 (time 1769078785.72 total 1322.45 last 72.88)
 - open/close 30678 (time 1769078795.72 total 1332.45 last 67.79)
 - open/close 31272 (time 1769078805.73 total 1342.46 last 59.35)
 - open/close 31917 (time 1769078815.73 total 1352.46 last 64.46)
 - open/close 32619 (time 1769078825.74 total 1362.47 last 70.17)
 - open/close 33226 (time 1769078835.76 total 1372.49 last 60.58)
 - open/close 33877 (time 1769078845.76 total 1382.50 last 65.05)
 - open/close 34379 (time 1769078855.78 total 1392.51 last 50.14)
 - open/close 34874 (time 1769078865.79 total 1402.52 last 49.44)
 - open/close 35478 (time 1769078875.79 total 1412.53 last 60.36)
 - open/close 36045 (time 1769078885.80 total 1422.53 last 56.69)
 - open/close 36578 (time 1769078895.80 total 1432.53 last 53.26)
 - open/close 37118 (time 1769078905.81 total 1442.54 last 53.98)
 - open/close 37642 (time 1769078915.82 total 1452.55 last 52.35)
 - open/close 38138 (time 1769078925.82 total 1462.55 last 49.58)
 - open/close 38691 (time 1769078935.83 total 1472.56 last 55.27)
 - open/close 39185 (time 1769078945.83 total 1482.56 last 49.39)
 - open/close 39737 (time 1769078955.84 total 1492.57 last 55.13)
 - open/close 40000 (time 1769078960.00 total 1496.73 last 63.30)
 - open/close 40535 (time 1769078970.00 total 1506.74 last 53.45)
 - open/close 41166 (time 1769078980.01 total 1516.74 last 63.07)
 - open/close 41802 (time 1769078990.01 total 1526.75 last 63.58)
 - open/close 42344 (time 1769079000.03 total 1536.76 last 54.13)
 - open/close 42894 (time 1769079010.07 total 1546.80 last 54.77)
 - open/close 43558 (time 1769079020.07 total 1556.80 last 66.39)
 - open/close 44234 (time 1769079030.08 total 1566.81 last 67.53)
 - open/close 44810 (time 1769079040.08 total 1576.82 last 57.57)
 - open/close 45423 (time 1769079050.09 total 1586.82 last 61.27)
 - open/close 46081 (time 1769079060.12 total 1596.85 last 65.59)
 - open/close 46686 (time 1769079070.12 total 1606.86 last 60.47)
 - open/close 47284 (time 1769079080.14 total 1616.87 last 59.73)
 - open/close 47856 (time 1769079090.15 total 1626.88 last 57.15)
 - open/close 48455 (time 1769079100.15 total 1636.88 last 59.88)
 - open/close 49131 (time 1769079110.16 total 1646.89 last 67.56)
 - open/close 49763 (time 1769079120.17 total 1656.90 last 63.14)
total: 50000 open/close in 1660.15 seconds: 30.12 ops/second
 - unlinked 0 (time 1769079128 ; total 0 ; last 0)
 - unlinked 10000 (time 1769079282 ; total 154 ; last 154)
 - unlinked 20000 (time 1769079441 ; total 313 ; last 159)
 - unlinked 30000 (time 1769079577 ; total 449 ; last 136)
 - unlinked 40000 (time 1769079723 ; total 595 ; last 146)
total: 50000 unlinks in 742 seconds: 67.385445 unlinks/second
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
PASS 101a (2527s)

== conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 06:05:07 (1769079907)
start mds service on oleg241-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
fail_loc=0x80002107
fail_val=20
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
oleg241-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99b88697e000.ost_server_uuid 50
osc.lustre-OST0000-osc-ffff99b88697e000.ost_server_uuid in FULL state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
PASS 101b (143s)

 SKIP: conf-sanity test_102 skipping excluded test 102

== conf-sanity test 103: rename filesystem name ========== 06:07:32 (1769080052)
Checking servers environments
Checking clients oleg241-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Starting client oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Started clients oleg241-client.virtnet: 
192.168.202.141@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff99b888581000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff99b888581000.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
disable quota as required
oleg241-server: Pool lustre.pool1 created
oleg241-server: Pool lustre.lustre created
oleg241-server: OST lustre-OST0000_UUID added to pool lustre.lustre
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:)
Stopping client oleg241-client.virtnet /mnt/lustre opts:
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
rename lustre to mylustre
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20

   Permanent disk data:
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20
  lustre:svname=mylustre-MDT0000
cmd: zfs set mountpoint=/tmp/mntfH07Nw lustre-mdt1/mdt1 && zfs set canmount=on lustre-mdt1/mdt1 && zfs mount lustre-mdt1/mdt1
cmd: zfs umount lustre-mdt1/mdt1 && zfs set canmount=off lustre-mdt1/mdt1 && zfs set mountpoint=none lustre-mdt1/mdt1
Writing lustre-mdt1/mdt1 properties
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:sys.timeout=20
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=mylustre
  lustre:svname=mylustre-MDT0000
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=mylustre-OST0000
cmd: zfs set mountpoint=/tmp/mnt8PFHXa lustre-ost1/ost1 && zfs set canmount=on lustre-ost1/ost1 && zfs mount lustre-ost1/ost1
cmd: zfs umount lustre-ost1/ost1 && zfs set canmount=off lustre-ost1/ost1 && zfs set mountpoint=none lustre-ost1/ost1
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.141@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=mylustre
  lustre:svname=mylustre-OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.141@tcp

   Permanent disk data:
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.141@tcp autodegrade=on
  lustre:svname=mylustre-OST0001
cmd: zfs set mountpoint=/tmp/mntXSbWuY lustre-ost2/ost2 && zfs set canmount=on lustre-ost2/ost2 && zfs mount lustre-ost2/ost2
cmd: zfs umount lustre-ost2/ost2 && zfs set canmount=off lustre-ost2/ost2 && zfs set mountpoint=none lustre-ost2/ost2
Writing lustre-ost2/ost2 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.141@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=mylustre
  lustre:svname=mylustre-OST0001
Checking servers environments
Checking clients oleg241-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  mylustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started mylustre-MDT0000
Start ost1: mount -t lustre -o localrecov  mylustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-mylustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started mylustre-OST0000
Start ost2: mount -t lustre -o localrecov  mylustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-mylustre-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started mylustre-OST0001
mount mylustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/mylustre /mnt/lustre
Starting client oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/mylustre /mnt/lustre
Started clients oleg241-client.virtnet: 
192.168.202.141@tcp:/mylustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.mylustre-OST0000-osc-ffff99b8893c9800.idle_timeout=debug
osc.mylustre-OST0001-osc-ffff99b8893c9800.idle_timeout=debug
disable quota as required
  File: /mnt/lustre/d103.conf-sanity/test-framework.sh
  Size: 310450    	Blocks: 1027       IO Block: 4194304 regular file
Device: c3aa56ceh/3282720462d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-01-22 06:08:53.000000000 -0500
Modify: 2026-01-22 06:08:54.000000000 -0500
Change: 2026-01-22 06:08:54.000000000 -0500
 Birth: 2026-01-22 06:08:53.000000000 -0500
Pool: mylustre.pool1
Pool: mylustre.lustre
mylustre-OST0000_UUID
mylustre-OST0000_UUID
oleg241-server: OST mylustre-OST0001_UUID added to pool mylustre.lustre
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:)
Stopping client oleg241-client.virtnet /mnt/lustre opts:
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
rename mylustre to tfs
checking for existing Lustre data: found

   Read previous values:
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20

   Permanent disk data:
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20
  lustre:svname=tfs-MDT0000
cmd: zfs set mountpoint=/tmp/mnt9oeeKT mylustre-mdt1/mdt1 && zfs set canmount=on mylustre-mdt1/mdt1 && zfs mount mylustre-mdt1/mdt1
cmd: zfs umount mylustre-mdt1/mdt1 && zfs set canmount=off mylustre-mdt1/mdt1 && zfs set mountpoint=none mylustre-mdt1/mdt1
Writing mylustre-mdt1/mdt1 properties
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:sys.timeout=20
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=tfs
  lustre:svname=tfs-MDT0000
checking for existing Lustre data: found

   Read previous values:
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=tfs-OST0000
cmd: zfs set mountpoint=/tmp/mntn4AAK3 mylustre-ost1/ost1 && zfs set canmount=on mylustre-ost1/ost1 && zfs mount mylustre-ost1/ost1
cmd: zfs umount mylustre-ost1/ost1 && zfs set canmount=off mylustre-ost1/ost1 && zfs set mountpoint=none mylustre-ost1/ost1
Writing mylustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.141@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=tfs
  lustre:svname=tfs-OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.141@tcp

   Permanent disk data:
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.141@tcp autodegrade=on
  lustre:svname=tfs-OST0001
cmd: zfs set mountpoint=/tmp/mntlWmViW mylustre-ost2/ost2 && zfs set canmount=on mylustre-ost2/ost2 && zfs mount mylustre-ost2/ost2
cmd: zfs umount mylustre-ost2/ost2 && zfs set canmount=off mylustre-ost2/ost2 && zfs set mountpoint=none mylustre-ost2/ost2
Writing mylustre-ost2/ost2 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.141@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=tfs
  lustre:svname=tfs-OST0001
Checking servers environments
Checking clients oleg241-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  tfs-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started tfs-MDT0000
Start ost1: mount -t lustre -o localrecov  tfs-ost1/ost1 /mnt/lustre-ost1
seq.cli-tfs-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started tfs-OST0000
Start ost2: mount -t lustre -o localrecov  tfs-ost2/ost2 /mnt/lustre-ost2
seq.cli-tfs-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started tfs-OST0001
mount tfs  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/tfs /mnt/lustre
Starting client oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/tfs /mnt/lustre
Started clients oleg241-client.virtnet: 
192.168.202.141@tcp:/tfs on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.tfs-OST0000-osc-ffff99b898bd2800.idle_timeout=debug
osc.tfs-OST0001-osc-ffff99b898bd2800.idle_timeout=debug
disable quota as required
  File: /mnt/lustre/d103.conf-sanity/test-framework.sh
  Size: 310450    	Blocks: 1027       IO Block: 4194304 regular file
Device: 32e2fa5ah/853736026d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-01-22 06:08:53.000000000 -0500
Modify: 2026-01-22 06:08:54.000000000 -0500
Change: 2026-01-22 06:08:54.000000000 -0500
 Birth: 2026-01-22 06:08:53.000000000 -0500
Pool: tfs.pool1
Pool: tfs.lustre
tfs-OST0000_UUID
tfs-OST0001_UUID
tfs-OST0000_UUID
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:)
Stopping client oleg241-client.virtnet /mnt/lustre opts:
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
rename tfs to lustre
checking for existing Lustre data: found

   Read previous values:
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20
  lustre:svname=lustre-MDT0000
cmd: zfs set mountpoint=/tmp/mntgaGX4s tfs-mdt1/mdt1 && zfs set canmount=on tfs-mdt1/mdt1 && zfs mount tfs-mdt1/mdt1
cmd: zfs umount tfs-mdt1/mdt1 && zfs set canmount=off tfs-mdt1/mdt1 && zfs set mountpoint=none tfs-mdt1/mdt1
Writing tfs-mdt1/mdt1 properties
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:sys.timeout=20
  lustre:version=1
  lustre:flags=5
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000
checking for existing Lustre data: found

   Read previous values:
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
cmd: zfs set mountpoint=/tmp/mnt3YR4SF tfs-ost1/ost1 && zfs set canmount=on tfs-ost1/ost1 && zfs mount tfs-ost1/ost1
cmd: zfs umount tfs-ost1/ost1 && zfs set canmount=off tfs-ost1/ost1 && zfs set mountpoint=none tfs-ost1/ost1
Writing tfs-ost1/ost1 properties
  lustre:mgsnode=192.168.202.141@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.141@tcp

   Permanent disk data:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.141@tcp autodegrade=on
  lustre:svname=lustre-OST0001
cmd: zfs set mountpoint=/tmp/mnt11WUuB tfs-ost2/ost2 && zfs set canmount=on tfs-ost2/ost2 && zfs mount tfs-ost2/ost2
cmd: zfs umount tfs-ost2/ost2 && zfs set canmount=off tfs-ost2/ost2 && zfs set mountpoint=none tfs-ost2/ost2
Writing tfs-ost2/ost2 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.141@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=2
  lustre:index=1
  lustre:fsname=lustre
  lustre:svname=lustre-OST0001
Checking servers environments
Checking clients oleg241-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Starting client oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
Started clients oleg241-client.virtnet: 
192.168.202.141@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff99b88a071000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff99b88a071000.idle_timeout=debug
disable quota as required
PASS 103 (348s)

== conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 06:13:21 (1769080401)
mountfsopt: acl,user_xattr
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg241-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
oleg241-server: oleg241-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
Start mds1: mount -t lustre -o localrecov,noacl  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
setfacl: /mnt/lustre: Operation not supported
PASS 104a (99s)

== conf-sanity test 104b: Mount uses last flock argument ========================================================== 06:15:00 (1769080500)
mount lustre with opts flock,localflock on /mnt/lustre3.....
Starting client: oleg241-client.virtnet:  -o flock,localflock 192.168.202.141@tcp:/lustre /mnt/lustre3
192.168.202.141@tcp:/lustre on /mnt/lustre3 type lustre (rw,checksum,encrypt,noflock,localflock,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose)
umount lustre on /mnt/lustre3.....
Stopping client oleg241-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock on /mnt/lustre3.....
Starting client: oleg241-client.virtnet:  -o localflock,flock 192.168.202.141@tcp:/lustre /mnt/lustre3
192.168.202.141@tcp:/lustre on /mnt/lustre3 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose)
umount lustre on /mnt/lustre3.....
Stopping client oleg241-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock,noflock on /mnt/lustre3.....
Starting client: oleg241-client.virtnet:  -o localflock,flock,noflock 192.168.202.141@tcp:/lustre /mnt/lustre3
umount lustre on /mnt/lustre3.....
Stopping client oleg241-client.virtnet /mnt/lustre3 (opts:)
PASS 104b (8s)

== conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 06:15:08 (1769080508)
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: rmmod: ERROR: Module mdd is in use
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
oleg241-server: oleg241-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg241-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
touch: cannot touch '/tmp/d105.conf-sanity/f105.conf-sanity': Read-only file system
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 105 (124s)

 SKIP: conf-sanity test_106 skipping SLOW test 106

== conf-sanity test 107: Unknown config param should not fail target mounting ========================================================== 06:17:13 (1769080633)
start mds service on oleg241-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
PASS 107 (130s)

 SKIP: conf-sanity test_108a skipping ALWAYS excluded test 108a

== conf-sanity test 108b: migrate from ZFS to ldiskfs ==== 06:19:24 (1769080764)

 SKIP: conf-sanity test_108b ldiskfs only test
SKIP 108b (1s)

== conf-sanity test 109a: test lctl clear_conf fsname ==== 06:19:25 (1769080765)

 SKIP: conf-sanity test_109a LU-8727: no implementation for ZFS
SKIP 109a (1s)

== conf-sanity test 109b: test lctl clear_conf one config ========================================================== 06:19:26 (1769080766)

 SKIP: conf-sanity test_109b LU-8727: no implementation for ZFS
SKIP 109b (2s)

== conf-sanity test 110: Adding large_dir with 3-level htree ========================================================== 06:19:28 (1769080768)

 SKIP: conf-sanity test_110 Only applicable to ldiskfs-based MDTs
SKIP 110 (1s)

 SKIP: conf-sanity test_111 skipping SLOW test 111

 SKIP: conf-sanity test_112a skipping ALWAYS excluded test 112a

 SKIP: conf-sanity test_112b skipping ALWAYS excluded test 112b

 SKIP: conf-sanity test_113 skipping ALWAYS excluded test 113

 SKIP: conf-sanity test_114 skipping SLOW test 114

 SKIP: conf-sanity test_115 skipping excluded test 115

== conf-sanity test 116: big size MDT support ============ 06:19:33 (1769080773)

 SKIP: conf-sanity test_116 ldiskfs only test
SKIP 116 (2s)

 SKIP: conf-sanity test_117 skipping ALWAYS excluded test 117

 SKIP: conf-sanity test_119 skipping ALWAYS excluded test 119

== conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 06:19:36 (1769080776)

 SKIP: conf-sanity test_120 mdt count < 2
SKIP 120 (2s)
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2
oleg241-server: oleg241-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg241-server: Force libcfs to create 2 CPU partitions
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
== conf-sanity test complete, duration 6694 sec ========== 06:20:08 (1769080808)
=== conf-sanity: start cleanup 06:20:09 (1769080809) ===
=== conf-sanity: finish cleanup 06:20:10 (1769080810) ===