-----============= acceptance-small: conf-sanity-1 ============----- Sun Dec 15 04:31:36 EST 2024
mgs: CentOS Linux release 7.9.2009 (Core)
MGS_OS_VERSION_ID=7
MGS_OS_ID=centos
MGS_OS_VERSION_CODE=117440512
MGS_OS_ID_LIKE=rhel fedora centos
mds1: CentOS Linux release 7.9.2009 (Core)
MDS1_OS_ID_LIKE=rhel fedora centos
MDS1_OS_ID=centos
MDS1_OS_VERSION_ID=7
MDS1_OS_VERSION_CODE=117440512
ost1: CentOS Linux release 7.9.2009 (Core)
OST1_OS_VERSION_CODE=117440512
OST1_OS_VERSION_ID=7
OST1_OS_ID_LIKE=rhel fedora centos
OST1_OS_ID=centos
client: CentOS Linux release 7.9.2009 (Core)
CLIENT_OS_ID=centos
CLIENT_OS_ID_LIKE=rhel fedora centos
CLIENT_OS_VERSION_ID=7
CLIENT_OS_VERSION_CODE=117440512
oleg160-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity-1.*ex: No such file or directory
excepting tests: 
skipping tests SLOW=no: 106
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg160-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server

== conf-sanity-1 test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 04:32:42 (1734255162)

 SKIP: conf-sanity-1 test_17 ldiskfs only test
SKIP 17 (1s)

== conf-sanity-1 test 18: check mkfs creates large journals ========================================================== 04:32:43 (1734255163)

 SKIP: conf-sanity-1 test_18 ldiskfs only test
SKIP 18 (1s)

== conf-sanity-1 test 21c: start mds between two osts, stop mds last ========================================================== 04:32:44 (1734255164)
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost2 service on oleg160-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop ost2 service on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 21c (115s)

== conf-sanity-1 test 29: permanently remove an OST ====== 04:34:39 (1734255279)
start ost2 service on oleg160-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Setting lustre-OST0001.osc.active from 1 to 0
Waiting 90s for '0'
Updated after 2s: want '0' got '0'
Live client success: got lustre-OST0001_UUID	FULL	DEACTIVATED
osc.lustre-OST0001-osc-MDT0000.active updated after 0 sec (got 0)
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
New client success: got 'lustre-OST0001_UUID	NEW	DEACTIVATED'
Setting lustre-OST0001.osc.active from 0 to 1
Waiting 90s for '1'
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 29 (99s)

== conf-sanity-1 test 35a: Reconnect to the last active server first ========================================================== 04:36:18 (1734255378)
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
debug=ha
Set up a fake failnode for the MDS
Wait for RECONNECT_INTERVAL seconds (10s)
conf-sanity.sh test_35a 2024-12-15 4h36m45s
Stopping the MDT: lustre-MDT0000
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
Restarting the MDT: lustre-MDT0000
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
Wait for df (23795) ... 
done
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout
Debug log: 72 lines, 72 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 35a (88s)

== conf-sanity-1 test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:37:46 (1734255466)
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
debug=ha
conf-sanity.sh test_35b 2024-12-15 4h38m02s
Set up a fake failnode for the MDS
at_max=0
at_max=0
Injecting EBUSY on MDS
fail_loc=0x80000136
mdc.lustre-MDT0000-mdc-ffff8800a9dd8800.stats=clear
Creating a test file and stat it
  File: '/mnt/lustre/d35b.conf-sanity-1/f35b.conf-sanity-1'
  Size: 0         	Blocks: 1          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d	Inode: 144115238826934274  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:38:46.000000000 -0500
Modify: 2024-12-15 04:38:46.000000000 -0500
Change: 2024-12-15 04:38:46.000000000 -0500
 Birth: -
Stop injecting EBUSY on MDS
fail_loc=0
done
at_max=600
at_max=600
Debug log: 48 lines, 48 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 35b (113s)

== conf-sanity-1 test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:39:39 (1734255579)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov -o nosvc -n  lustre-mdt1/mdt1 /mnt/lustre-mds1
nomtab: 1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
Start lustre-mdt1/mdt1 without service
Commit the device label on lustre-mdt1/mdt1
Started lustre:MDT0000
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg160-server
Starting mds1: -o localrecov -o nomgs,force  lustre-mdt1/mdt1 /mnt/lustre-mds1
force: 1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
blah blah
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 41b (66s)

== conf-sanity-1 test 46a: handle ost additional - wide striped file ========================================================== 04:40:45 (1734255645)
Testing with 2 OSTs
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg160-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9f08000.ost_server_uuid 50
osc.lustre-OST0001-osc-ffff8800a9f08000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre2.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre2
/mnt/lustre2
stripe_count:  -1 stripe_size:   4194304 pattern:       raid0 stripe_offset: -1

/mnt/lustre2/f41b.conf-sanity-1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	             2	          0x2	   0x240000400

/mnt/lustre2/widestripe
lmm_stripe_count:  2
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 1
	obdidx		 objid		 objid		 group
	     1	             2	          0x2	   0x280000400
	     0	            34	         0x22	   0x240000400

  File: '/mnt/lustre/widestripe'
  Size: 3         	Blocks: 2          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115238826934273  Links: 1
Access: (0674/-rw-rwxr--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:41:11.000000000 -0500
Modify: 2024-12-15 04:41:11.000000000 -0500
Change: 2024-12-15 04:41:12.000000000 -0500
 Birth: -
umount lustre on /mnt/lustre2.....
Stopping client oleg160-client.virtnet /mnt/lustre2 (opts:)
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
stop ost1 service on oleg160-server
stop mds service on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 46a (122s)

== conf-sanity-1 test 47a: server restart does not lose lru_resize ========================================================== 04:42:47 (1734255767)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
setup single mount lustre success
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012e088000.lru_size=100
ldlm.namespaces.lustre-OST0000-osc-ffff88012e088000.lru_size=100
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012e088000.lru_size=100
Failing ost1 on oleg160-server
Stopping /mnt/lustre-ost1 (opts:) on oleg160-server
04:43:37 (1734255817) shut down
facet: ost1 facet_host: oleg160-server facet_failover_host: oleg160-server
Failover ost1 to oleg160-server
mount facets: ost1
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
04:43:50 (1734255830) targets are mounted
04:43:50 (1734255830) facet_failover done
Failing mds1 on oleg160-server
Stopping /mnt/lustre-mds1 (opts:) on oleg160-server
04:43:51 (1734255831) shut down
facet: mds1 facet_host: oleg160-server facet_failover_host: oleg160-server
Failover mds1 to oleg160-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
04:44:04 (1734255844) targets are mounted
04:44:04 (1734255844) facet_failover done
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012e088000.lru_size=100
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
PASS 47a (97s)

== conf-sanity-1 test 47b: client restart does not lose lru_resize ========================================================== 04:44:24 (1734255864)
Failing mds1 on oleg160-server
04:44:26 (1734255866) shut down
facet: mds1 facet_host: oleg160-server facet_failover_host: oleg160-server
Failover mds1 to oleg160-server
mount facets: mds1
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
04:44:50 (1734255890) targets are mounted
04:44:50 (1734255890) facet_failover done
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 95
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 95
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
setup single mount lustre success
ln: failed to create symbolic link '/home/green/git/lustre-release/lustre/utils/lctl': File exists

 SKIP: conf-sanity-1 test_47b Can not create symlink /usr/sbin/lctl
SKIP 47b (74s)

== conf-sanity-1 test 48: too many acls on file ========== 04:45:39 (1734255939)
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 17
Start of lustre-mdt1/mdt1 on mds1 failed 17
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg160-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of lustre-ost1/ost1 on ost1 failed 17
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
mount.lustre: according to /etc/mtab 192.168.201.160@tcp:/lustre is already mounted on /mnt/lustre
setup single mount lustre success
debug=0
debug=0
/mnt/lustre
stripe_count:  -1 stripe_size:   4194304 pattern:       raid0 stripe_offset: -1

/mnt/lustre/widestripe
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	             4	          0x4	   0x240000400

It is expected to hold at least 4500 ACL entries
  File: '/mnt/lustre/widestripe'
  Size: 3         	Blocks: 1          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115205272502275  Links: 1
Access: (0664/-rw-rw-r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:45:44.000000000 -0500
Modify: 2024-12-15 04:45:44.000000000 -0500
Change: 2024-12-15 04:47:02.000000000 -0500
 Birth: -
getfacl: Removing leading '/' from absolute path names
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg160-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 48 (148s)

== conf-sanity-1 test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:48:07 (1734256087)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
setup single mount lustre success
check ldlm_timout...
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 49a (47s)

== conf-sanity-1 test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:48:54 (1734256134)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
PASS 49b (41s)

== conf-sanity-1 test 50c: lazystatfs one server down ==== 04:49:35 (1734256175)
start mds service on oleg160-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg160-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
llite.lustre-ffff88012e3bc000.lazystatfs=1
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
oleg160-server: oleg160-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7021
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-OST0001_UUID       382976        3072      377856   1% /mnt/lustre[OST:1]

filesystem_summary:       382976        3072      377856   1% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (5729) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 50c (94s)

== conf-sanity-1 test 50d: lazystatfs client/server conn race ========================================================== 04:51:09 (1734256269)
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg160-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800aaa3f800.lazystatfs=1
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7021
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-OST0001_UUID       382976        3072      377856   1% /mnt/lustre[OST:1]

filesystem_summary:       382976        3072      377856   1% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (10677) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/intel_rapl/refcnt': 'No such device'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 50d (89s)

== conf-sanity-1 test 50f: normal statfs one server in down ========================================================== 04:52:38 (1734256358)
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
start ost2 service on oleg160-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
oleg160-server: oleg160-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
llite.lustre-ffff88012a56f800.lazystatfs=0
multiop /mnt/lustre v_f
TMPPIPE=/tmp/multiop_open_wait_pipe.7021
start ost2 service on oleg160-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0001
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 50f (116s)

== conf-sanity-1 test 50g: deactivated OST should not cause panic ========================================================== 04:54:34 (1734256474)
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
start ost2 service on oleg160-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg160-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b3dae800.ost_server_uuid 50
osc.lustre-OST0001-osc-ffff8800b3dae800.ost_server_uuid in FULL state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
Filesystem                  1K-blocks  Used Available Use% Mounted on
192.168.201.160@tcp:/lustre    382976  3072    377856   1% /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 50g (78s)

== conf-sanity-1 test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:55:52 (1734256552)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
setup single mount lustre success
fail_loc=0x142
start ost2 service on oleg160-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec
stop ost2 service on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 51 (129s)

== conf-sanity-1 test 54a: test llverdev and partial verify of device ========================================================== 04:58:01 (1734256681)

 SKIP: conf-sanity-1 test_54a ldiskfs only test
SKIP 54a (1s)

== conf-sanity-1 test 55: check lov_objid size =========== 04:58:02 (1734256682)

 SKIP: conf-sanity-1 test_55 ldiskfs only test
SKIP 55 (1s)

== conf-sanity-1 test 56a: check big OST indexes and out-of-index-order start ========================================================== 04:58:03 (1734256683)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2

   Permanent disk data:
Target:     lustre:OST2710
Index:      10000
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=10000
  lustre:fsname=lustre
  lustre:svname=lustre:OST2710

   Permanent disk data:
Target:     lustre:OST03e8
Index:      1000
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost2 /dev/vdf
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost2/ost2
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost2/ost2 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=1000
  lustre:fsname=lustre
  lustre:svname=lustre:OST03e8
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST2710-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST2710
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg160-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST03e8-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST03e8
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
ok
OBDS:
1000: lustre-OST03e8_UUID ACTIVE
10000: lustre-OST2710_UUID ACTIVE
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.00631743 s, 648 kB/s
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:)
Stopping client oleg160-client.virtnet /mnt/lustre opts:
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 56a (116s)

== conf-sanity-1 test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:59:59 (1734256799)

 SKIP: conf-sanity-1 test_56b needs >= 3 MDTs
SKIP 56b (2s)

== conf-sanity-1 test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:00:01 (1734256801)
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp autodegrade=on sys.timeout=20 failover.node=192.168.201.160@tcp autodegrade=on
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:failover.node=192.168.201.160@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg160-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Cannot assign requested address
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 99
oleg160-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg160-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
Start of lustre-ost1/ost1 on ost1 failed 99
umount lustre on /mnt/lustre.....
stop ost1 service on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
PASS 57a (76s)

== conf-sanity-1 test 57b: initial registration from servicenode should not fail ========================================================== 05:01:17 (1734256877)
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x1042
              (OST update no_primnode )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp autodegrade=on sys.timeout=20 failover.node=192.168.201.160@tcp autodegrade=on
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:failover.node=192.168.201.160@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=4162
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
PASS 57b (96s)

== conf-sanity-1 test 59: writeconf mount option ========= 05:02:53 (1734256973)
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
original ost count: 1 (expect > 0)
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
after mdt writeconf count: 0 (expect 0)
OST start without writeconf should fail:
oleg160-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: No such file or directory
oleg160-server: Is the MGS specification correct?
oleg160-server: Is the filesystem name correct?
oleg160-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg160-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
OST start with writeconf should succeed:
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
after ost writeconf count: 1 (expect 1)
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
after ost2 writeconf count: 2 (expect 2)
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 59 (125s)

== conf-sanity-1 test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 05:04:58 (1734257098)

 SKIP: conf-sanity-1 test_60a ldiskfs only test
SKIP 60a (2s)

== conf-sanity-1 test 62: start with disabled journal ==== 05:05:00 (1734257100)

 SKIP: conf-sanity-1 test_62 ldiskfs only test
SKIP 62 (1s)

== conf-sanity-1 test 64: check lfs df --lazy ============ 05:05:01 (1734257101)
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg160-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
stop ost2 service on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
/home/green/git/lustre-release/lustre/utils/lfs df
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3200      378368   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID       382976        3072      377856   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID : Transport endpoint is not connected

filesystem_summary:       382976        3072      377856   1% /mnt/lustre

umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg160-server: 
oleg160-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg160-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 64 (109s)

== conf-sanity-1 test 66: replace nids =================== 05:06:50 (1734257210)
client=34615898 MDS=34615898 OSS=34615898
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
Setting lustre-OST0000.osc.active from 1 to 0
Waiting 90s for '0'
Updated after 6s: want '0' got '0'
replace_nids should fail if MDS, OSTs and clients are UP
oleg160-server: error: replace_nids: Operation now in progress
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 115
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
replace_nids should fail if MDS and OSTs are UP
oleg160-server: error: replace_nids: Operation now in progress
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 115
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
replace_nids should fail if MDS is UP
oleg160-server: error: replace_nids: Operation now in progress
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 115
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
start mds service on oleg160-server
Starting mds1: -o localrecov -o nosvc  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
Start lustre-mdt1/mdt1 without service
Started lustre-MDT0000
command should accept two parameters
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
correct device name should be passed
oleg160-server: error: replace_nids: No such device or address
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 6
wrong nids list should not destroy the system
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
replace OST nid
command should accept two parameters
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
wrong nids list should not destroy the system
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
set NIDs with failover
replace MDS nid
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Setting lustre-OST0000.osc.active from 0 to 1
Waiting 90s for '1'
Updated after 2s: want '1' got '1'
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 66 (140s)

== conf-sanity-1 test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 05:09:10 (1734257350)

 SKIP: conf-sanity-1 test_70e needs >= 2 MDTs
SKIP 70e (1s)

== conf-sanity-1 test 75: The order of --index should be irrelevant ========================================================== 05:09:11 (1734257351)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1/mdt1
  xattr=sa
  dnodesize=auto
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=69
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1/mdt1
  xattr=sa
  dnodesize=auto
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=69
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 75 (58s)

== conf-sanity-1 test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:10:09 (1734257409)

 SKIP: conf-sanity-1 test_78 ldiskfs only test
SKIP 78 (1s)

== conf-sanity-1 test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:10:10 (1734257410)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: 
oleg160-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg160-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 22
oleg160-server: 
oleg160-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg160-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 22
oleg160-server: 
oleg160-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg160-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 22
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 79 (59s)

== conf-sanity-1 test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 05:11:10 (1734257470)

 SKIP: conf-sanity-1 test_83 ldiskfs only test
SKIP 83 (2s)

== conf-sanity-1 test 86: Replacing mkfs.lustre -G option ========================================================== 05:11:11 (1734257471)

 SKIP: conf-sanity-1 test_86 LU-6442: no such mkfs params for ZFS OSTs
SKIP 86 (1s)

== conf-sanity-1 test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 05:11:12 (1734257472)

 SKIP: conf-sanity-1 test_87 ldiskfs only test
SKIP 87 (1s)

== conf-sanity-1 test 89: check tunefs --param and --erase-param{s} options ========================================================== 05:11:13 (1734257473)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:)
tunefs --param failover.node=192.0.2.254@tcp0
tunefs --param failover.node=192.0.2.255@tcp0
tunefs --erase-param failover.node
tunefs --erase-params
tunefs --param failover.node=192.0.2.254@tcp0 --erase-params
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 89 (52s)

== conf-sanity-1 test 93: register mulitple MDT at the same time ========================================================== 05:12:05 (1734257525)

 SKIP: conf-sanity-1 test_93 needs >= 3 MDTs
SKIP 93 (1s)

== conf-sanity-1 test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:12:06 (1734257526)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
setup single mount lustre success
lustre-MDT0000:
lustre-OST0000:
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
PASS 100 (49s)

== conf-sanity-1 test 104a: Make sure user defined options are reflected in mount ========================================================== 05:12:55 (1734257575)
mountfsopt: acl,user_xattr
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
Starting mds1: -o localrecov,noacl  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
setfacl: /mnt/lustre: Operation not supported
PASS 104a (44s)

== conf-sanity-1 test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:13:39 (1734257619)
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
oleg160-server: rmmod: ERROR: Module lustre is in use
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
touch: cannot touch '/tmp/d105.conf-sanity-1/f105.conf-sanity-1': Read-only file system
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
PASS 105 (76s)

== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:14:55 (1734257695)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        86842         329       86513   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID        16200         392       15808   3% /mnt/lustre[OST:0]

filesystem_summary:        16137         329       15808   3% /mnt/lustre

seq.cli-lustre-OST0000-super.width=0x1ffffff
fail_loc=0x1312
fail_val=5
 - open/close 1884 (time 1734257754.29 total 10.00 last 188.38)
 - open/close 4178 (time 1734257764.29 total 20.00 last 229.37)
 - open/close 6240 (time 1734257774.29 total 30.00 last 206.16)
 - open/close 8390 (time 1734257784.30 total 40.01 last 214.89)
 - open/close 10000 (time 1734257791.46 total 47.18 last 224.58)
 - open/close 12504 (time 1734257801.47 total 57.18 last 250.36)
 - open/close 14590 (time 1734257811.47 total 67.18 last 208.58)
 - open/close 16766 (time 1734257821.47 total 77.18 last 217.56)
 - open/close 19180 (time 1734257831.47 total 87.19 last 241.31)
 - open/close 20000 (time 1734257835.15 total 90.87 last 222.86)
 - open/close 22184 (time 1734257845.15 total 100.87 last 218.38)
 - open/close 24013 (time 1734257855.16 total 110.87 last 182.86)
 - open/close 25962 (time 1734257865.16 total 120.87 last 194.84)
 - open/close 27887 (time 1734257875.16 total 130.88 last 192.38)
 - open/close 29794 (time 1734257885.17 total 140.88 last 190.66)
 - open/close 32083 (time 1734257895.17 total 150.88 last 228.82)
 - open/close 34999 (time 1734257905.17 total 160.89 last 291.52)
 - open/close 36899 (time 1734257915.17 total 170.89 last 189.99)
 - open/close 38657 (time 1734257925.18 total 180.89 last 175.67)
 - open/close 40000 (time 1734257933.53 total 189.25 last 160.80)
 - open/close 41885 (time 1734257943.54 total 199.25 last 188.43)
 - open/close 44236 (time 1734257953.54 total 209.25 last 235.04)
 - open/close 46110 (time 1734257963.54 total 219.25 last 187.39)
 - open/close 47996 (time 1734257973.54 total 229.25 last 188.59)
 - open/close 49663 (time 1734257983.54 total 239.26 last 166.63)
 - open/close 51218 (time 1734257993.55 total 249.26 last 155.48)
 - open/close 52711 (time 1734258003.55 total 259.26 last 149.28)
 - open/close 54158 (time 1734258013.55 total 269.26 last 144.66)
 - open/close 55630 (time 1734258023.55 total 279.27 last 147.14)
 - open/close 57268 (time 1734258033.56 total 289.27 last 163.69)
 - open/close 58589 (time 1734258043.56 total 299.27 last 132.09)
 - open/close 59794 (time 1734258053.56 total 309.28 last 120.47)
 - open/close 61101 (time 1734258063.57 total 319.29 last 130.58)
 - open/close 62376 (time 1734258073.58 total 329.29 last 127.45)
 - open/close 63576 (time 1734258083.58 total 339.29 last 119.95)
total: 64768 open/close in 347.34 seconds: 186.47 ops/second
 - /unlink 2557 (time 1734258102.97 total 10.00 last 255.66)
 - /unlink 4881 (time 1734258112.97 total 20.00 last 232.38)
 - /unlink 7335 (time 1734258122.97 total 30.00 last 245.38)
 - /unlink 9896 (time 1734258132.97 total 40.01 last 255.94)
 - /unlink 12532 (time 1734258142.98 total 50.01 last 263.48)
 - /unlink 15039 (time 1734258152.98 total 60.02 last 250.65)
 - /unlink 17530 (time 1734258162.98 total 70.02 last 249.06)
 - /unlink 20000 (time 1734258172.32 total 79.36 last 264.49)
 - /unlink 23074 (time 1734258182.32 total 89.36 last 307.35)
 - /unlink 25830 (time 1734258192.32 total 99.36 last 275.54)
 - /unlink 28851 (time 1734258202.32 total 109.36 last 302.09)
 - /unlink 30000 (time 1734258206.12 total 113.15 last 302.80)
 - /unlink 33020 (time 1734258216.12 total 123.16 last 301.94)
 - /unlink 35412 (time 1734258226.12 total 133.16 last 239.16)
 - /unlink 37577 (time 1734258236.13 total 143.16 last 216.42)
 - /unlink 39636 (time 1734258246.13 total 153.16 last 205.86)
 - /unlink 41682 (time 1734258256.13 total 163.17 last 204.51)
 - /unlink 43765 (time 1734258266.14 total 173.17 last 208.22)
 - /unlink 45853 (time 1734258276.14 total 183.17 last 208.80)
 - /unlink 47855 (time 1734258286.14 total 193.17 last 200.17)
 - /unlink 49602 (time 1734258296.15 total 203.18 last 174.58)
 - /unlink 51789 (time 1734258306.15 total 213.19 last 218.60)
 - /unlink 53634 (time 1734258316.15 total 223.19 last 184.45)
 - /unlink 55463 (time 1734258326.16 total 233.19 last 182.84)
 - /unlink 57448 (time 1734258336.16 total 243.19 last 198.48)
 - /unlink 59486 (time 1734258346.16 total 253.20 last 203.68)
 - /unlink 60000 (time 1734258348.36 total 255.40 last 233.94)
 - /unlink 62688 (time 1734258358.36 total 265.40 last 268.72)
total: 64768 /unlink in 273.10 seconds: 237.16 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 1320 (time 1734258391.93 total 10.00 last 131.95)
 - open/close 2715 (time 1734258401.94 total 20.01 last 139.45)
 - open/close 4065 (time 1734258411.94 total 30.01 last 134.94)
 - open/close 5404 (time 1734258421.95 total 40.02 last 133.85)
 - open/close 6799 (time 1734258431.95 total 50.02 last 139.44)
 - open/close 8237 (time 1734258441.95 total 60.02 last 143.77)
 - open/close 9856 (time 1734258451.96 total 70.03 last 161.85)
 - open/close 11472 (time 1734258461.96 total 80.03 last 161.51)
 - open/close 13164 (time 1734258471.96 total 90.03 last 169.19)
 - open/close 14929 (time 1734258481.97 total 100.04 last 176.41)
 - open/close 16874 (time 1734258491.97 total 110.04 last 194.48)
 - open/close 18897 (time 1734258501.97 total 120.04 last 202.20)
 - open/close 20000 (time 1734258508.61 total 126.68 last 166.24)
 - open/close 21629 (time 1734258518.62 total 136.69 last 162.76)
 - open/close 23201 (time 1734258528.62 total 146.69 last 157.15)
 - open/close 24668 (time 1734258538.62 total 156.69 last 146.64)
 - open/close 26110 (time 1734258548.62 total 166.70 last 144.17)
 - open/close 27553 (time 1734258558.63 total 176.70 last 144.24)
 - open/close 29055 (time 1734258568.63 total 186.70 last 150.15)
 - open/close 30000 (time 1734258574.86 total 192.93 last 151.81)
 - open/close 31421 (time 1734258584.86 total 202.93 last 142.10)
 - open/close 32757 (time 1734258594.86 total 212.93 last 133.60)
 - open/close 34346 (time 1734258604.87 total 222.94 last 158.79)
 - open/close 35756 (time 1734258614.87 total 232.94 last 140.89)
 - open/close 37195 (time 1734258624.87 total 242.95 last 143.88)
 - open/close 38690 (time 1734258634.88 total 252.95 last 149.48)
 - open/close 40000 (time 1734258644.27 total 262.34 last 139.51)
 - open/close 41469 (time 1734258654.27 total 272.34 last 146.82)
 - open/close 43010 (time 1734258664.28 total 282.35 last 154.03)
 - open/close 44609 (time 1734258674.28 total 292.35 last 159.86)
 - open/close 46061 (time 1734258684.28 total 302.35 last 145.17)
 - open/close 47823 (time 1734258694.28 total 312.35 last 176.14)
 - open/close 49411 (time 1734258704.29 total 322.36 last 158.79)
 - open/close 50000 (time 1734258707.87 total 325.95 last 164.09)
 - open/close 51521 (time 1734258717.88 total 335.95 last 152.05)
 - open/close 52837 (time 1734258727.88 total 345.95 last 131.55)
 - open/close 54222 (time 1734258737.88 total 355.95 last 138.50)
 - open/close 55562 (time 1734258747.89 total 365.96 last 133.93)
 - open/close 56995 (time 1734258757.89 total 375.96 last 143.20)
 - open/close 58381 (time 1734258767.90 total 385.97 last 138.51)
 - open/close 59856 (time 1734258777.90 total 395.97 last 147.46)
 - open/close 61447 (time 1734258787.91 total 405.98 last 158.94)
 - open/close 63206 (time 1734258797.92 total 415.99 last 175.85)
total: 64768 open/close in 425.48 seconds: 152.22 ops/second
 - /unlink 2613 (time 1734258818.91 total 10.00 last 261.27)
 - /unlink 5948 (time 1734258828.91 total 20.00 last 333.49)
 - /unlink 8944 (time 1734258838.91 total 30.00 last 299.54)
 - /unlink 10000 (time 1734258842.40 total 33.49 last 302.89)
 - /unlink 12927 (time 1734258852.40 total 43.49 last 292.68)
 - /unlink 15611 (time 1734258862.40 total 53.49 last 268.39)
 - /unlink 18535 (time 1734258872.40 total 63.49 last 292.39)
 - /unlink 20000 (time 1734258877.57 total 68.66 last 283.35)
 - /unlink 22882 (time 1734258887.57 total 78.66 last 288.19)
 - /unlink 25847 (time 1734258897.57 total 88.67 last 296.40)
 - /unlink 28732 (time 1734258907.57 total 98.67 last 288.49)
 - /unlink 30000 (time 1734258911.91 total 103.00 last 292.34)
 - /unlink 32914 (time 1734258921.91 total 113.01 last 291.32)
 - /unlink 36118 (time 1734258931.91 total 123.01 last 320.37)
 - /unlink 39573 (time 1734258941.91 total 133.01 last 345.49)
 - /unlink 42973 (time 1734258951.91 total 143.01 last 339.94)
 - /unlink 46225 (time 1734258961.92 total 153.01 last 325.15)
 - /unlink 48902 (time 1734258971.92 total 163.02 last 267.52)
 - /unlink 50000 (time 1734258975.93 total 167.02 last 274.23)
 - /unlink 52759 (time 1734258985.93 total 177.02 last 275.89)
 - /unlink 55461 (time 1734258995.93 total 187.02 last 270.18)
 - /unlink 58414 (time 1734259005.93 total 197.02 last 295.26)
 - /unlink 60000 (time 1734259011.48 total 202.57 last 285.70)
 - /unlink 62980 (time 1734259021.48 total 212.58 last 297.96)
total: 64768 /unlink in 217.99 seconds: 297.12 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 1667 (time 1734259057.05 total 10.00 last 166.69)
 - open/close 3234 (time 1734259067.05 total 20.00 last 156.69)
 - open/close 4810 (time 1734259077.06 total 30.01 last 157.51)
 - open/close 6501 (time 1734259087.06 total 40.01 last 169.00)
 - open/close 8120 (time 1734259097.07 total 50.02 last 161.82)
 - open/close 9680 (time 1734259107.07 total 60.02 last 155.95)
 - open/close 10000 (time 1734259109.08 total 62.03 last 159.42)
 - open/close 11496 (time 1734259119.09 total 72.04 last 149.48)
 - open/close 12927 (time 1734259129.09 total 82.04 last 143.08)
 - open/close 14384 (time 1734259139.09 total 92.04 last 145.67)
 - open/close 15823 (time 1734259149.10 total 102.05 last 143.70)
 - open/close 17178 (time 1734259159.11 total 112.06 last 135.44)
 - open/close 18556 (time 1734259169.11 total 122.06 last 137.76)
 - open/close 19907 (time 1734259179.12 total 132.07 last 135.02)
 - open/close 21424 (time 1734259189.12 total 142.07 last 151.69)
 - open/close 22845 (time 1734259199.12 total 152.07 last 142.10)
 - open/close 24279 (time 1734259209.12 total 162.07 last 143.35)
 - open/close 25678 (time 1734259219.12 total 172.07 last 139.88)
 - open/close 27092 (time 1734259229.13 total 182.08 last 141.36)
 - open/close 28546 (time 1734259239.13 total 192.08 last 145.34)
 - open/close 30000 (time 1734259249.12 total 202.07 last 145.54)
 - open/close 31618 (time 1734259259.12 total 212.07 last 161.78)
 - open/close 33218 (time 1734259269.12 total 222.07 last 159.97)
 - open/close 34999 (time 1734259279.13 total 232.07 last 178.08)
 - open/close 36779 (time 1734259289.13 total 242.08 last 177.88)
 - open/close 38313 (time 1734259299.14 total 252.09 last 153.29)
 - open/close 39972 (time 1734259309.14 total 262.09 last 165.90)
 - open/close 41543 (time 1734259319.15 total 272.10 last 156.99)
 - open/close 43063 (time 1734259329.15 total 282.10 last 151.88)
 - open/close 44339 (time 1734259339.16 total 292.11 last 127.53)
 - open/close 45782 (time 1734259349.16 total 302.11 last 144.25)
 - open/close 47346 (time 1734259359.16 total 312.11 last 156.39)
 - open/close 48767 (time 1734259369.16 total 322.11 last 142.10)
 - open/close 50000 (time 1734259377.09 total 330.04 last 155.58)
 - open/close 51807 (time 1734259387.09 total 340.04 last 180.69)
 - open/close 53560 (time 1734259397.09 total 350.04 last 175.22)
 - open/close 55291 (time 1734259407.09 total 360.04 last 173.09)
 - open/close 57062 (time 1734259417.10 total 370.04 last 177.07)
 - open/close 58932 (time 1734259427.10 total 380.05 last 186.95)
 - open/close 60000 (time 1734259433.05 total 386.00 last 179.36)
 - open/close 61759 (time 1734259443.05 total 396.00 last 175.88)
 - open/close 63395 (time 1734259453.05 total 406.00 last 163.58)
total: 64768 open/close in 413.51 seconds: 156.63 ops/second
 - /unlink 3663 (time 1734259471.78 total 10.00 last 366.26)
 - /unlink 7368 (time 1734259481.79 total 20.00 last 370.43)
 - /unlink 10000 (time 1734259488.80 total 27.02 last 375.11)
 - /unlink 13904 (time 1734259498.80 total 37.02 last 390.31)
 - /unlink 17548 (time 1734259508.81 total 47.02 last 364.36)
 - /unlink 20000 (time 1734259515.00 total 53.22 last 395.56)
 - /unlink 23792 (time 1734259525.01 total 63.22 last 379.13)
 - /unlink 27433 (time 1734259535.01 total 73.22 last 364.09)
 - /unlink 30000 (time 1734259542.83 total 81.05 last 328.16)
 - /unlink 32754 (time 1734259552.83 total 91.05 last 275.26)
 - /unlink 35590 (time 1734259562.84 total 101.05 last 283.55)
 - /unlink 38519 (time 1734259572.84 total 111.06 last 292.81)
 - /unlink 40000 (time 1734259577.46 total 115.67 last 320.76)
 - /unlink 42866 (time 1734259587.46 total 125.68 last 286.49)
 - /unlink 45880 (time 1734259597.46 total 135.68 last 301.30)
 - /unlink 49035 (time 1734259607.46 total 145.68 last 315.45)
 - /unlink 50000 (time 1734259610.36 total 148.58 last 333.23)
 - /unlink 53483 (time 1734259620.36 total 158.58 last 348.23)
 - /unlink 56528 (time 1734259630.36 total 168.58 last 304.49)
 - /unlink 59889 (time 1734259640.36 total 178.58 last 336.08)
 - /unlink 63252 (time 1734259650.36 total 188.58 last 336.24)
total: 64768 /unlink in 192.91 seconds: 335.74 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 1840 (time 1734259683.75 total 10.01 last 183.83)
 - open/close 3837 (time 1734259693.76 total 20.01 last 199.60)
 - open/close 5666 (time 1734259703.76 total 30.01 last 182.90)
 - open/close 7461 (time 1734259713.76 total 40.02 last 179.43)
 - open/close 9046 (time 1734259723.76 total 50.02 last 158.47)
 - open/close 10000 (time 1734259729.67 total 55.93 last 161.48)
 - open/close 11763 (time 1734259739.67 total 65.93 last 176.26)
 - open/close 13724 (time 1734259749.68 total 75.94 last 195.97)
 - open/close 15601 (time 1734259759.68 total 85.94 last 187.66)
 - open/close 17387 (time 1734259769.68 total 95.94 last 178.58)
 - open/close 19601 (time 1734259779.69 total 105.95 last 221.30)
 - open/close 21748 (time 1734259789.69 total 115.95 last 214.66)
 - open/close 23960 (time 1734259799.69 total 125.95 last 221.14)
 - open/close 25967 (time 1734259809.69 total 135.95 last 200.68)
 - open/close 28134 (time 1734259819.70 total 145.95 last 216.67)
 - open/close 30000 (time 1734259829.04 total 155.30 last 199.68)
 - open/close 31767 (time 1734259839.05 total 165.31 last 176.53)
 - open/close 33592 (time 1734259849.05 total 175.31 last 182.46)
 - open/close 35579 (time 1734259859.06 total 185.31 last 198.64)
 - open/close 37250 (time 1734259869.06 total 195.32 last 167.02)
 - open/close 38904 (time 1734259879.06 total 205.32 last 165.40)
 - open/close 40000 (time 1734259884.78 total 211.04 last 191.59)
 - open/close 41960 (time 1734259894.78 total 221.04 last 196.00)
 - open/close 43872 (time 1734259904.78 total 231.04 last 191.18)
 - open/close 45998 (time 1734259914.78 total 241.04 last 212.58)
 - open/close 48271 (time 1734259924.78 total 251.04 last 227.30)
 - open/close 50000 (time 1734259932.79 total 259.04 last 216.07)
 - open/close 52256 (time 1734259942.79 total 269.04 last 225.58)
 - open/close 54470 (time 1734259952.79 total 279.04 last 221.38)
 - open/close 56874 (time 1734259962.79 total 289.05 last 240.31)
 - open/close 59042 (time 1734259972.80 total 299.05 last 216.70)
 - open/close 60000 (time 1734259976.80 total 303.06 last 238.96)
 - open/close 62213 (time 1734259986.81 total 313.06 last 221.23)
 - open/close 64531 (time 1734259996.87 total 323.13 last 230.38)
total: 64768 open/close in 324.08 seconds: 199.85 ops/second
 - /unlink 4534 (time 1734260008.86 total 10.00 last 453.32)
 - /unlink 8782 (time 1734260018.86 total 20.00 last 424.77)
 - /unlink 10000 (time 1734260021.56 total 22.71 last 450.52)
 - /unlink 14332 (time 1734260031.56 total 32.71 last 433.12)
 - /unlink 18423 (time 1734260041.56 total 42.71 last 408.99)
 - /unlink 20000 (time 1734260045.55 total 46.69 last 395.81)
 - /unlink 24066 (time 1734260055.55 total 56.70 last 406.58)
 - /unlink 27759 (time 1734260065.55 total 66.70 last 369.23)
 - /unlink 30000 (time 1734260072.47 total 73.61 last 324.00)
 - /unlink 33537 (time 1734260082.47 total 83.61 last 353.67)
 - /unlink 37298 (time 1734260092.47 total 93.61 last 376.08)
 - /unlink 40000 (time 1734260099.70 total 100.84 last 373.77)
 - /unlink 43766 (time 1734260109.70 total 110.84 last 376.58)
 - /unlink 47596 (time 1734260119.70 total 120.85 last 382.95)
 - /unlink 50000 (time 1734260125.85 total 127.00 last 390.69)
 - /unlink 54234 (time 1734260135.85 total 137.00 last 423.38)
 - /unlink 58759 (time 1734260145.85 total 147.00 last 452.44)
 - /unlink 60000 (time 1734260149.02 total 150.17 last 391.92)
total: 64768 /unlink in 159.26 seconds: 406.68 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 3227 (time 1734260188.97 total 10.00 last 322.69)
 - open/close 6560 (time 1734260198.98 total 20.00 last 333.16)
 - open/close 9890 (time 1734260208.98 total 30.01 last 332.94)
 - open/close 13152 (time 1734260218.98 total 40.01 last 326.14)
 - open/close 16592 (time 1734260228.98 total 50.01 last 343.96)
 - open/close 19860 (time 1734260238.98 total 60.01 last 326.76)
 - open/close 22876 (time 1734260248.98 total 70.01 last 301.56)
 - open/close 26078 (time 1734260258.99 total 80.01 last 320.12)
 - open/close 29310 (time 1734260268.99 total 90.02 last 323.17)
 - open/close 30000 (time 1734260271.25 total 92.28 last 304.63)
 - open/close 33535 (time 1734260281.25 total 102.28 last 353.47)
 - open/close 36980 (time 1734260291.25 total 112.28 last 344.47)
 - open/close 40000 (time 1734260299.90 total 120.93 last 349.39)
 - open/close 42835 (time 1734260309.90 total 130.93 last 283.40)
 - open/close 45315 (time 1734260319.91 total 140.93 last 247.91)
 - open/close 48028 (time 1734260329.91 total 150.93 last 271.28)
 - open/close 50000 (time 1734260337.21 total 158.24 last 269.95)
 - open/close 52625 (time 1734260347.21 total 168.24 last 262.46)
 - open/close 55724 (time 1734260357.21 total 178.24 last 309.85)
 - open/close 59318 (time 1734260367.22 total 188.24 last 359.39)
 - open/close 60000 (time 1734260369.44 total 190.46 last 307.12)
 - open/close 63036 (time 1734260379.44 total 200.47 last 303.53)
total: 64768 open/close in 206.28 seconds: 313.98 ops/second
 - /unlink 5716 (time 1734260396.22 total 10.00 last 571.53)
 - /unlink 10000 (time 1734260402.60 total 16.39 last 671.04)
 - /unlink 15916 (time 1734260412.60 total 26.39 last 591.51)
 - /unlink 20000 (time 1734260420.65 total 34.44 last 507.21)
 - /unlink 25646 (time 1734260430.65 total 44.44 last 564.59)
 - /unlink 30000 (time 1734260439.60 total 53.38 last 486.71)
 - /unlink 34364 (time 1734260449.60 total 63.39 last 436.29)
 - /unlink 39200 (time 1734260459.60 total 73.39 last 483.56)
 - /unlink 44073 (time 1734260469.60 total 83.39 last 487.22)
 - /unlink 48398 (time 1734260479.60 total 93.39 last 432.49)
 - /unlink 50000 (time 1734260482.69 total 96.48 last 519.11)
 - /unlink 54624 (time 1734260492.69 total 106.48 last 462.36)
 - /unlink 59248 (time 1734260502.69 total 116.48 last 462.33)
total: 64768 /unlink in 126.46 seconds: 512.18 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
fail_loc=0
fail_val=0
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
PASS 106 (2853s)

== conf-sanity-1 test 109a: test lctl clear_conf fsname == 06:02:28 (1734260548)

 SKIP: conf-sanity-1 test_109a LU-8727: no implementation for ZFS
SKIP 109a (0s)

== conf-sanity-1 test 109b: test lctl clear_conf one config ========================================================== 06:02:28 (1734260548)

 SKIP: conf-sanity-1 test_109b LU-8727: no implementation for ZFS
SKIP 109b (1s)

== conf-sanity-1 test 116: big size MDT support ========== 06:02:29 (1734260549)

 SKIP: conf-sanity-1 test_116 ldiskfs only test
SKIP 116 (1s)

== conf-sanity-1 test 122a: Check OST sequence update ==== 06:02:30 (1734260550)

 SKIP: conf-sanity-1 test_122a needs >= 2 MDTs
SKIP 122a (1s)

== conf-sanity-1 test 123ah: del_ost cancels config log entries correctly ========================================================== 06:02:31 (1734260551)
Checking servers environments
Checking clients oleg160-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
Starting client oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
Started clients oleg160-client.virtnet: 
192.168.201.160@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800a9f32000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800a9f32000.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 4s: want 'procname_uid' got 'procname_uid'
disable quota as required
del_ost: dry run for target lustre-OST0000
config_log: lustre-MDT0000
[DRY RUN] cancel catalog 'lustre-MDT0000:20':"- { index: 20, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-MDT0000:19':"- { index: 19, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.201.160@tcp }"
[DRY RUN] cancel catalog 'lustre-MDT0000:18':"- { index: 18, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-client
[DRY RUN] cancel catalog 'lustre-client:25':"- { index: 25, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-client:24':"- { index: 24, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.201.160@tcp }"
[DRY RUN] cancel catalog 'lustre-client:23':"- { index: 23, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-MDT0000
cancel catalog lustre-MDT0000 log_idx 20: done
cancel catalog lustre-MDT0000 log_idx 19: done
cancel catalog lustre-MDT0000 log_idx 18: done
del_ost: cancelled 3 catalog entries
config_log: lustre-client
cancel catalog lustre-client log_idx 25: done
cancel catalog lustre-client log_idx 24: done
cancel catalog lustre-client log_idx 23: done
del_ost: cancelled 3 catalog entries
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
oleg160-server: rmmod: ERROR: Module lustre is in use
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
fail_loc=0
PASS 123ah (92s)

== conf-sanity-1 test 132: hsm_actions processed after failover ========================================================== 06:04:04 (1734260644)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 mdt.hsm_control=enabled
Writing lustre-mdt1/mdt1 properties
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:sys.timeout=20
  lustre:mdt.hsm_control=enabled
  lustre:version=1
  lustre:flags=69
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
PASS 132 (26s)

== conf-sanity-1 test 133: stripe QOS: free space balance in a pool ========================================================== 06:04:30 (1734260670)

 SKIP: conf-sanity-1 test_133 needs >= 4 OSTs
SKIP 133 (1s)

== conf-sanity-1 test 135: check the behavior when changelog is wrapped around ========================================================== 06:04:31 (1734260671)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
oleg160-client: fail_loc=0x1312
oleg160-client: fail_val=5
oleg160-server: fail_loc=0x1312
oleg160-server: fail_val=5
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
Registered 1 changelog users: 'cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 3178 (time 1734260710.06 total 10.00 last 317.77)
total: 4500 open/close in 14.00 seconds: 321.42 ops/second
total: 4500 /unlink in 7.54 seconds: 596.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 3382 (time 1734260733.45 total 10.00 last 338.16)
total: 4500 open/close in 13.50 seconds: 333.26 ops/second
total: 4500 /unlink in 6.74 seconds: 667.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 3515 (time 1734260755.56 total 10.00 last 351.46)
total: 4500 open/close in 12.41 seconds: 362.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
total: 4500 /unlink in 5.79 seconds: 777.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 4084 (time 1734260775.51 total 10.00 last 408.34)
total: 4500 open/close in 11.34 seconds: 396.71 ops/second
total: 4500 /unlink in 5.74 seconds: 783.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 3898 (time 1734260794.46 total 10.00 last 389.73)
total: 4500 open/close in 11.33 seconds: 397.31 ops/second
total: 4500 /unlink in 4.76 seconds: 946.34 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 10973: 22787 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
PASS 135 (156s)

== conf-sanity-1 test 136: don't panic with bad obdecho setup ========================================================== 06:07:07 (1734260827)

 SKIP: conf-sanity-1 test_136 needs >= 2 MDTs
SKIP 136 (1s)

== conf-sanity-1 test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:07:08 (1734260828)

 SKIP: conf-sanity-1 test_137 needs >= 3 MDTs
SKIP 137 (1s)

== conf-sanity-1 test 140: remove_updatelog script actions ========================================================== 06:07:09 (1734260829)

 SKIP: conf-sanity-1 test_140 needs >= 2 MDTs
SKIP 140 (1s)

== conf-sanity-1 test 151a: damaged local config doesn't prevent mounting ========================================================== 06:07:10 (1734260830)

 SKIP: conf-sanity-1 test_151a ldiskfs only test
SKIP 151a (0s)

== conf-sanity-1 test 151b: -ENOSPC doesn't affect mount ========================================================== 06:07:10 (1734260830)

 SKIP: conf-sanity-1 test_151b ldiskfs only test
SKIP 151b (1s)

== conf-sanity-1 test 153a: bypass invalid NIDs quickly == 06:07:11 (1734260831)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
waiting for mount ...
waiting for mount ...
          "192.168.201.160@tcp": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 5 }
          "192.168.252.112@tcp": { connects: 1, replied: 0, uptodate: discovering, sec_ago: 10 }
          "10.252.252.113@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
          "192.168.201.160@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
setup single mount lustre success
umount lustre on /mnt/lustre.....
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
PASS 153a (178s)

== conf-sanity-1 test 153b: added IPv6 NID support ======= 06:10:09 (1734261009)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
Checking servers environments
Checking clients oleg160-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
Starting client oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
Started clients oleg160-client.virtnet: 
192.168.201.160@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800aaa3d800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800aaa3d800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 7s: want 'procname_uid' got 'procname_uid'
disable quota as required
PASS 153b (72s)

== conf-sanity-1 test 153c: don't stuck on unreached NID ========================================================== 06:11:21 (1734261081)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
start mds service on oleg160-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg160-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
          "192.168.252.112@tcp": { connects: 3, replied: 0, uptodate: discovering, sec_ago: 0 }
          "192.168.252.113@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 10 }
          "192.168.252.115@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 5 }
connection attempts: 7
Waiting for mount to fail
mount.lustre: mount 192.168.252.112@tcp:192.168.252.113@tcp:192.168.252.115@tcp:/lustre at /mnt/lustre failed: Input/output error
Is the MGS running?
umount lustre on /mnt/lustre.....
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server
unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
PASS 153c (320s)

== conf-sanity-1 test 154: expand .. on rename after MDT backup restore ========================================================== 06:16:41 (1734261401)

 SKIP: conf-sanity-1 test_154 ldiskfs only test
SKIP 154 (1s)
Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
oleg160-server: oleg160-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
== conf-sanity-1 test complete, duration 6324 sec ======== 06:17:00 (1734261420)
=== conf-sanity-1: start cleanup 06:17:00 (1734261420) ===
=== conf-sanity-1: finish cleanup 06:17:01 (1734261421) ===