-----============= acceptance-small: conf-sanity-1 ============----- Sun Dec 15 04:31:54 EST 2024
mgs: CentOS Linux release 7.9.2009 (Core)
MGS_OS_VERSION_ID=7
MGS_OS_ID=centos
MGS_OS_VERSION_CODE=117440512
MGS_OS_ID_LIKE=rhel fedora centos
mds1: CentOS Linux release 7.9.2009 (Core)
MDS1_OS_ID_LIKE=rhel fedora centos
MDS1_OS_ID=centos
MDS1_OS_VERSION_ID=7
MDS1_OS_VERSION_CODE=117440512
ost1: CentOS Linux release 7.9.2009 (Core)
OST1_OS_VERSION_CODE=117440512
OST1_OS_VERSION_ID=7
OST1_OS_ID_LIKE=rhel fedora centos
OST1_OS_ID=centos
client: CentOS Linux release 7.9.2009 (Core)
CLIENT_OS_ID=centos
CLIENT_OS_ID_LIKE=rhel fedora centos
CLIENT_OS_VERSION_ID=7
CLIENT_OS_VERSION_CODE=117440512
oleg136-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity-1.*ex: No such file or directory
excepting tests: 
skipping tests SLOW=no: 106
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg136-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server

== conf-sanity-1 test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 04:33:15 (1734255195)

 SKIP: conf-sanity-1 test_17 ldiskfs only test
SKIP 17 (0s)

== conf-sanity-1 test 18: check mkfs creates large journals ========================================================== 04:33:16 (1734255196)

 SKIP: conf-sanity-1 test_18 ldiskfs only test
SKIP 18 (0s)

== conf-sanity-1 test 21c: start mds between two osts, stop mds last ========================================================== 04:33:16 (1734255196)
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost2 service on oleg136-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop ost2 service on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 21c (148s)

== conf-sanity-1 test 29: permanently remove an OST ====== 04:35:44 (1734255344)
start ost2 service on oleg136-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Setting lustre-OST0001.osc.active from 1 to 0
Waiting 90s for '0'
Live client success: got lustre-OST0001_UUID	FULL	DEACTIVATED
Waiting 30 secs for osc.lustre-OST0001-osc-MDT0000.active
osc.lustre-OST0001-osc-MDT0000.active updated after 5 sec (got 0)
osc.lustre-OST0001-osc-MDT0001.active updated after 5 sec (got 0)
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
New client success: got 'lustre-OST0001_UUID	NEW	DEACTIVATED'
Setting lustre-OST0001.osc.active from 0 to 1
Waiting 90s for '1'
Updated after 2s: want '1' got '1'
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 29 (121s)

== conf-sanity-1 test 35a: Reconnect to the last active server first ========================================================== 04:37:45 (1734255465)
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
debug=ha
Set up a fake failnode for the MDS
Wait for RECONNECT_INTERVAL seconds (10s)
conf-sanity.sh test_35a 2024-12-15 4h38m15s
Stopping the MDT: lustre-MDT0000
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
Restarting the MDT: lustre-MDT0000
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
Wait for df (29191) ... 
done
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout
Debug log: 80 lines, 80 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 35a (107s)

== conf-sanity-1 test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:39:32 (1734255572)
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
debug=ha
conf-sanity.sh test_35b 2024-12-15 4h39m52s
Set up a fake failnode for the MDS
at_max=0
at_max=0
Injecting EBUSY on MDS
fail_loc=0x80000136
mdc.lustre-MDT0000-mdc-ffff8800a9d35800.stats=clear
mdc.lustre-MDT0001-mdc-ffff8800a9d35800.stats=clear
Creating a test file and stat it
  File: '/mnt/lustre/d35b.conf-sanity-1/f35b.conf-sanity-1'
  Size: 0         	Blocks: 1          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d	Inode: 144115238826934274  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:40:36.000000000 -0500
Modify: 2024-12-15 04:40:36.000000000 -0500
Change: 2024-12-15 04:40:36.000000000 -0500
 Birth: -
Stop injecting EBUSY on MDS
fail_loc=0
done
at_max=600
at_max=600
Debug log: 76 lines, 76 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 35b (179s)

== conf-sanity-1 test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:42:31 (1734255751)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov -o nosvc -n  lustre-mdt1/mdt1 /mnt/lustre-mds1
nomtab: 1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
Start lustre-mdt1/mdt1 without service
Commit the device label on lustre-mdt1/mdt1
Started lustre:MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg136-server
Starting mds1: -o localrecov -o nomgs,force  lustre-mdt1/mdt1 /mnt/lustre-mds1
force: 1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
blah blah
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
PASS 41b (108s)

== conf-sanity-1 test 46a: handle ost additional - wide striped file ========================================================== 04:44:19 (1734255859)
Testing with 2 OSTs
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff880135c52000.ost_server_uuid 50
osc.lustre-OST0001-osc-ffff880135c52000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre2.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre2
/mnt/lustre2
stripe_count:  -1 stripe_size:   4194304 pattern:       raid0 stripe_offset: -1

/mnt/lustre2/f41b.conf-sanity-1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	             2	          0x2	   0x280000400

/mnt/lustre2/widestripe
lmm_stripe_count:  2
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	            34	         0x22	   0x280000400
	     1	             2	          0x2	   0x2c0000401

  File: '/mnt/lustre/widestripe'
  Size: 3         	Blocks: 2          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115238826934273  Links: 1
Access: (0674/-rw-rwxr--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:44:49.000000000 -0500
Modify: 2024-12-15 04:44:49.000000000 -0500
Change: 2024-12-15 04:44:50.000000000 -0500
 Birth: -
umount lustre on /mnt/lustre2.....
Stopping client oleg136-client.virtnet /mnt/lustre2 (opts:)
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
stop ost1 service on oleg136-server
stop mds service on oleg136-server
stop mds service on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 46a (144s)

== conf-sanity-1 test 47a: server restart does not lose lru_resize ========================================================== 04:46:43 (1734256003)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
setup single mount lustre success
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800abe21800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff8800abe21800.lru_size=100
ldlm.namespaces.lustre-OST0000-osc-ffff8800abe21800.lru_size=100
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800abe21800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff8800abe21800.lru_size=100
Failing ost1 on oleg136-server
Stopping /mnt/lustre-ost1 (opts:) on oleg136-server
04:47:27 (1734256047) shut down
facet: ost1 facet_host: oleg136-server facet_failover_host: oleg136-server
Failover ost1 to oleg136-server
mount facets: ost1
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
04:47:41 (1734256061) targets are mounted
04:47:41 (1734256061) facet_failover done
Failing mds1 on oleg136-server
Stopping /mnt/lustre-mds1 (opts:) on oleg136-server
04:47:42 (1734256062) shut down
facet: mds1 facet_host: oleg136-server facet_failover_host: oleg136-server
Failover mds1 to oleg136-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
04:47:55 (1734256075) targets are mounted
04:47:55 (1734256075) facet_failover done
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800abe21800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff8800abe21800.lru_size=100
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
PASS 47a (100s)

== conf-sanity-1 test 47b: client restart does not lose lru_resize ========================================================== 04:48:23 (1734256103)
Failing mds1 on oleg136-server
04:48:24 (1734256104) shut down
facet: mds1 facet_host: oleg136-server facet_failover_host: oleg136-server
Failover mds1 to oleg136-server
mount facets: mds1
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
04:48:46 (1734256126) targets are mounted
04:48:46 (1734256126) facet_failover done
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 95
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 95
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
setup single mount lustre success
ln: failed to create symbolic link '/home/green/git/lustre-release/lustre/utils/lctl': File exists

 SKIP: conf-sanity-1 test_47b Can not create symlink /usr/sbin/lctl
SKIP 47b (71s)

== conf-sanity-1 test 48: too many acls on file ========== 04:49:34 (1734256174)
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 17
Start of lustre-mdt1/mdt1 on mds1 failed 17
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg136-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of lustre-ost1/ost1 on ost1 failed 17
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
mount.lustre: according to /etc/mtab 192.168.201.136@tcp:/lustre is already mounted on /mnt/lustre
setup single mount lustre success
debug=0
debug=0
/mnt/lustre
stripe_count:  -1 stripe_size:   4194304 pattern:       raid0 stripe_offset: -1

/mnt/lustre/widestripe
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	             4	          0x4	   0x280000401

It is expected to hold at least 4500 ACL entries
  File: '/mnt/lustre/widestripe'
  Size: 3         	Blocks: 1          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115205289279491  Links: 1
Access: (0664/-rw-rw-r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:49:38.000000000 -0500
Modify: 2024-12-15 04:49:38.000000000 -0500
Change: 2024-12-15 04:50:36.000000000 -0500
 Birth: -
getfacl: Removing leading '/' from absolute path names
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg136-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 48 (145s)

== conf-sanity-1 test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:51:59 (1734256319)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
setup single mount lustre success
check ldlm_timout...
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 49a (57s)

== conf-sanity-1 test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:52:56 (1734256376)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
PASS 49b (57s)

== conf-sanity-1 test 50c: lazystatfs one server down ==== 04:53:53 (1734256433)
start mds service on oleg136-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg136-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800aa44d800.lazystatfs=1
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
oleg136-server: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7778
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID       383616        3072      378496   1% /mnt/lustre[MDT:1]
lustre-OST0001_UUID       382976        4096      376832   2% /mnt/lustre[OST:1]

filesystem_summary:       382976        4096      376832   2% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (21687) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 50c (133s)

== conf-sanity-1 test 50d: lazystatfs client/server conn race ========================================================== 04:56:06 (1734256566)
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg136-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800a9d7b800.lazystatfs=1
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7778
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID       383616        3072      378496   1% /mnt/lustre[MDT:1]
lustre-OST0001_UUID       382976        4096      376832   2% /mnt/lustre[OST:1]

filesystem_summary:       382976        4096      376832   2% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (28194) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 50d (103s)

== conf-sanity-1 test 50f: normal statfs one server in down ========================================================== 04:57:49 (1734256669)
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
start ost2 service on oleg136-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
oleg136-server: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800a86b9800.lazystatfs=0
multiop /mnt/lustre v_f
TMPPIPE=/tmp/multiop_open_wait_pipe.7778
start ost2 service on oleg136-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0001
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 50f (172s)

== conf-sanity-1 test 50g: deactivated OST should not cause panic ========================================================== 05:00:41 (1734256841)
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
start ost2 service on oleg136-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9d36000.ost_server_uuid 50
osc.lustre-OST0001-osc-ffff8800a9d36000.ost_server_uuid in FULL state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
Filesystem                  1K-blocks  Used Available Use% Mounted on
192.168.201.136@tcp:/lustre    382976  4096    376832   2% /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 50g (144s)

== conf-sanity-1 test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:03:05 (1734256985)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
setup single mount lustre success
fail_loc=0x142
start ost2 service on oleg136-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 51 (199s)

== conf-sanity-1 test 54a: test llverdev and partial verify of device ========================================================== 05:06:24 (1734257184)

 SKIP: conf-sanity-1 test_54a ldiskfs only test
SKIP 54a (1s)

== conf-sanity-1 test 55: check lov_objid size =========== 05:06:26 (1734257186)

 SKIP: conf-sanity-1 test_55 ldiskfs only test
SKIP 55 (1s)

== conf-sanity-1 test 56a: check big OST indexes and out-of-index-order start ========================================================== 05:06:27 (1734257187)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2

   Permanent disk data:
Target:     lustre:OST2710
Index:      10000
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.136@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=10000
  lustre:fsname=lustre
  lustre:svname=lustre:OST2710

   Permanent disk data:
Target:     lustre:OST03e8
Index:      1000
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost2 /dev/vdf
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost2/ost2
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost2/ost2 properties
  lustre:mgsnode=192.168.201.136@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=1000
  lustre:fsname=lustre
  lustre:svname=lustre:OST03e8
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST2710-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST2710
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg136-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST03e8-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST03e8
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
ok
OBDS:
1000: lustre-OST03e8_UUID ACTIVE
10000: lustre-OST2710_UUID ACTIVE
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.0075041 s, 546 kB/s
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:)
Stopping client oleg136-client.virtnet /mnt/lustre opts:
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 56a (150s)

== conf-sanity-1 test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:08:57 (1734257337)

 SKIP: conf-sanity-1 test_56b needs >= 3 MDTs
SKIP 56b (1s)

== conf-sanity-1 test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:08:58 (1734257338)
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 autodegrade=on failover.node=192.168.201.136@tcp autodegrade=on
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.136@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:failover.node=192.168.201.136@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg136-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Cannot assign requested address
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 99
oleg136-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg136-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
Start of lustre-ost1/ost1 on ost1 failed 99
umount lustre on /mnt/lustre.....
stop ost1 service on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
PASS 57a (100s)

== conf-sanity-1 test 57b: initial registration from servicenode should not fail ========================================================== 05:10:38 (1734257438)
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.136@tcp

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x1042
              (OST update no_primnode )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.136@tcp failover.node=192.168.201.136@tcp autodegrade=on
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.201.136@tcp
  lustre:failover.node=192.168.201.136@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=4162
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
PASS 57b (122s)

== conf-sanity-1 test 59: writeconf mount option ========= 05:12:40 (1734257560)
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
original ost count: 1 (expect > 0)
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
after mdt writeconf count: 0 (expect 0)
OST start without writeconf should fail:
oleg136-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: No such file or directory
oleg136-server: Is the MGS specification correct?
oleg136-server: Is the filesystem name correct?
oleg136-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg136-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
OST start with writeconf should succeed:
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
after ost writeconf count: 1 (expect 1)
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
after ost2 writeconf count: 2 (expect 2)
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 59 (150s)

== conf-sanity-1 test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 05:15:10 (1734257710)

 SKIP: conf-sanity-1 test_60a ldiskfs only test
SKIP 60a (1s)

== conf-sanity-1 test 62: start with disabled journal ==== 05:15:11 (1734257711)

 SKIP: conf-sanity-1 test_62 ldiskfs only test
SKIP 62 (1s)

== conf-sanity-1 test 64: check lfs df --lazy ============ 05:15:12 (1734257712)
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg136-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
stop ost2 service on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
/home/green/git/lustre-release/lustre/utils/lfs df
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID       383616        3072      378496   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID       382976        4096      376832   2% /mnt/lustre[OST:0]
lustre-OST0001_UUID : Transport endpoint is not connected

filesystem_summary:       382976        4096      376832   2% /mnt/lustre

umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg136-server: 
oleg136-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg136-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 64 (141s)

== conf-sanity-1 test 66: replace nids =================== 05:17:33 (1734257853)
client=34615898 MDS=34615898 OSS=34615898
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
Setting lustre-OST0000.osc.active from 1 to 0
Waiting 90s for '0'
Updated after 4s: want '0' got '0'
replace_nids should fail if MDS, OSTs and clients are UP
oleg136-server: error: replace_nids: Operation now in progress
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 115
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
replace_nids should fail if MDS and OSTs are UP
oleg136-server: error: replace_nids: Operation now in progress
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 115
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
replace_nids should fail if MDS is UP
oleg136-server: error: replace_nids: Operation now in progress
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 115
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
start mds service on oleg136-server
Starting mds1: -o localrecov -o nosvc  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
Start lustre-mdt1/mdt1 without service
Started lustre-MDT0000
command should accept two parameters
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
correct device name should be passed
oleg136-server: error: replace_nids: No such device or address
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 6
wrong nids list should not destroy the system
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
replace OST nid
command should accept two parameters
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
wrong nids list should not destroy the system
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
set NIDs with failover
replace MDS nid
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Setting lustre-OST0000.osc.active from 0 to 1
Waiting 90s for '1'
Waiting 80s for '1'
Updated after 23s: want '1' got '1'
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 66 (236s)

== conf-sanity-1 test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 05:21:29 (1734258089)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1/mdt1
  xattr=sa
  dnodesize=auto
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=101
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:MDT0000
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.136@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid

   Permanent disk data:
Target:     lustre:MDT0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -f -O canmount=off lustre-mdt2 /dev/vdd
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt2/mdt2
  xattr=sa
  dnodesize=auto
Writing lustre-mdt2/mdt2 properties
  lustre:mgsnode=192.168.201.136@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=97
  lustre:index=1
  lustre:fsname=lustre
  lustre:svname=lustre:MDT0001
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-server: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50
oleg136-server: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50
oleg136-server: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
umount lustre on /mnt/lustre.....
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
PASS 70e (107s)

== conf-sanity-1 test 75: The order of --index should be irrelevant ========================================================== 05:23:17 (1734258197)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1/mdt1
  xattr=sa
  dnodesize=auto
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=69
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.136@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1/mdt1
  xattr=sa
  dnodesize=auto
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=69
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.136@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 75 (116s)

== conf-sanity-1 test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:25:12 (1734258312)

 SKIP: conf-sanity-1 test_78 ldiskfs only test
SKIP 78 (2s)

== conf-sanity-1 test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:25:14 (1734258314)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
oleg136-server: 
oleg136-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg136-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22
oleg136-server: 
oleg136-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg136-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22
oleg136-server: 
oleg136-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg136-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 79 (102s)

== conf-sanity-1 test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 05:26:56 (1734258416)

 SKIP: conf-sanity-1 test_83 ldiskfs only test
SKIP 83 (2s)

== conf-sanity-1 test 86: Replacing mkfs.lustre -G option ========================================================== 05:26:58 (1734258418)

 SKIP: conf-sanity-1 test_86 LU-6442: no such mkfs params for ZFS OSTs
SKIP 86 (1s)

== conf-sanity-1 test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 05:26:59 (1734258419)

 SKIP: conf-sanity-1 test_87 ldiskfs only test
SKIP 87 (2s)

== conf-sanity-1 test 89: check tunefs --param and --erase-param{s} options ========================================================== 05:27:01 (1734258421)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:)
tunefs --param failover.node=192.0.2.254@tcp0
tunefs --param failover.node=192.0.2.255@tcp0
tunefs --erase-param failover.node
tunefs --erase-params
tunefs --param failover.node=192.0.2.254@tcp0 --erase-params
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 89 (87s)

== conf-sanity-1 test 93: register mulitple MDT at the same time ========================================================== 05:28:28 (1734258508)

 SKIP: conf-sanity-1 test_93 needs >= 3 MDTs
SKIP 93 (1s)

== conf-sanity-1 test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:28:29 (1734258509)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
setup single mount lustre success
lustre-MDT0000:
lustre-OST0000:
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
PASS 100 (87s)

== conf-sanity-1 test 104a: Make sure user defined options are reflected in mount ========================================================== 05:29:56 (1734258596)
mountfsopt: acl,user_xattr
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
Starting mds1: -o localrecov,noacl  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
Starting mds2: -o localrecov,noacl  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
setfacl: /mnt/lustre: Operation not supported
PASS 104a (71s)

== conf-sanity-1 test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:31:07 (1734258667)
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
oleg136-server: rmmod: ERROR: Module lustre is in use
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
touch: cannot touch '/tmp/d105.conf-sanity-1/f105.conf-sanity-1': Read-only file system
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
PASS 105 (127s)

== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:33:14 (1734258794)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        86890         331       86559   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        87086         312       86774   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        16266         458       15808   3% /mnt/lustre[OST:0]

filesystem_summary:        16451         643       15808   4% /mnt/lustre

seq.cli-lustre-OST0000-super.width=0x1ffffff
fail_loc=0x1312
fail_val=5
 - open/close 1487 (time 1734258870.00 total 10.00 last 148.66)
 - open/close 2908 (time 1734258880.00 total 20.01 last 142.01)
 - open/close 4391 (time 1734258890.01 total 30.01 last 148.23)
 - open/close 5906 (time 1734258900.01 total 40.02 last 151.42)
 - open/close 7309 (time 1734258910.02 total 50.02 last 140.24)
 - open/close 8789 (time 1734258920.02 total 60.02 last 147.97)
 - open/close 10000 (time 1734258928.02 total 68.02 last 151.39)
 - open/close 11650 (time 1734258938.02 total 78.03 last 164.94)
 - open/close 13332 (time 1734258948.03 total 88.03 last 168.14)
 - open/close 14923 (time 1734258958.03 total 98.03 last 159.04)
 - open/close 16323 (time 1734258968.03 total 108.04 last 139.96)
 - open/close 17736 (time 1734258978.03 total 118.04 last 141.27)
 - open/close 19148 (time 1734258988.04 total 128.04 last 141.16)
 - open/close 20000 (time 1734258993.96 total 133.96 last 143.91)
 - open/close 21427 (time 1734259003.96 total 143.97 last 142.63)
 - open/close 22883 (time 1734259013.97 total 153.97 last 145.50)
 - open/close 24397 (time 1734259023.97 total 163.98 last 151.34)
 - open/close 25866 (time 1734259033.98 total 173.98 last 146.84)
 - open/close 27208 (time 1734259043.98 total 183.98 last 134.19)
 - open/close 28588 (time 1734259053.98 total 193.98 last 137.99)
 - open/close 30000 (time 1734259063.95 total 203.95 last 141.66)
 - open/close 31322 (time 1734259073.95 total 213.95 last 132.19)
 - open/close 32739 (time 1734259083.95 total 223.95 last 141.68)
 - open/close 34155 (time 1734259093.95 total 233.96 last 141.58)
 - open/close 35466 (time 1734259103.96 total 243.96 last 131.04)
 - open/close 36792 (time 1734259113.96 total 253.96 last 132.56)
 - open/close 38077 (time 1734259123.97 total 263.97 last 128.37)
 - open/close 39314 (time 1734259133.97 total 273.98 last 123.64)
 - open/close 40000 (time 1734259139.20 total 279.21 last 131.24)
 - open/close 41261 (time 1734259149.20 total 289.21 last 126.06)
 - open/close 42428 (time 1734259159.21 total 299.22 last 116.59)
 - open/close 43687 (time 1734259169.22 total 309.22 last 125.82)
 - open/close 44932 (time 1734259179.22 total 319.23 last 124.44)
 - open/close 46279 (time 1734259189.23 total 329.23 last 134.63)
 - open/close 47534 (time 1734259199.23 total 339.24 last 125.44)
 - open/close 48823 (time 1734259209.23 total 349.24 last 128.90)
 - open/close 50000 (time 1734259218.74 total 358.74 last 123.84)
 - open/close 51248 (time 1734259228.75 total 368.75 last 124.69)
 - open/close 52498 (time 1734259238.75 total 378.76 last 124.96)
 - open/close 53812 (time 1734259248.75 total 388.76 last 131.40)
 - open/close 55197 (time 1734259258.76 total 398.76 last 138.45)
 - open/close 56539 (time 1734259268.76 total 408.76 last 134.15)
 - open/close 58023 (time 1734259278.76 total 418.76 last 148.39)
 - open/close 59617 (time 1734259288.76 total 428.77 last 159.34)
 - open/close 60000 (time 1734259291.59 total 431.60 last 135.30)
 - open/close 61404 (time 1734259301.59 total 441.60 last 140.40)
 - open/close 62816 (time 1734259311.60 total 451.61 last 141.07)
 - open/close 64138 (time 1734259321.61 total 461.62 last 132.09)
total: 64768 open/close in 466.36 seconds: 138.88 ops/second
 - /unlink 2082 (time 1734259338.11 total 10.00 last 208.15)
 - /unlink 4341 (time 1734259348.11 total 20.00 last 225.89)
 - /unlink 6735 (time 1734259358.11 total 30.01 last 239.35)
 - /unlink 8919 (time 1734259368.11 total 40.01 last 218.38)
 - /unlink 10000 (time 1734259373.05 total 44.95 last 218.80)
 - /unlink 12556 (time 1734259383.05 total 54.95 last 255.59)
 - /unlink 15204 (time 1734259393.05 total 64.95 last 264.78)
 - /unlink 17803 (time 1734259403.05 total 74.95 last 259.87)
 - /unlink 20000 (time 1734259411.50 total 83.39 last 260.20)
 - /unlink 22795 (time 1734259421.50 total 93.39 last 279.49)
 - /unlink 25535 (time 1734259431.50 total 103.40 last 273.94)
 - /unlink 28162 (time 1734259441.50 total 113.40 last 262.61)
 - /unlink 30000 (time 1734259449.23 total 121.12 last 237.95)
 - /unlink 32708 (time 1734259459.23 total 131.13 last 270.74)
 - /unlink 35732 (time 1734259469.23 total 141.13 last 302.34)
 - /unlink 38840 (time 1734259479.23 total 151.13 last 310.79)
 - /unlink 40000 (time 1734259483.11 total 155.00 last 299.43)
 - /unlink 43181 (time 1734259493.11 total 165.00 last 318.02)
 - /unlink 46211 (time 1734259503.11 total 175.01 last 302.94)
 - /unlink 49433 (time 1734259513.11 total 185.01 last 322.11)
 - /unlink 52545 (time 1734259523.12 total 195.01 last 311.14)
 - /unlink 55593 (time 1734259533.12 total 205.02 last 304.66)
 - /unlink 58258 (time 1734259543.12 total 215.02 last 266.43)
 - /unlink 60000 (time 1734259550.54 total 222.44 last 234.86)
 - /unlink 62411 (time 1734259560.54 total 232.44 last 241.10)
total: 64768 /unlink in 241.69 seconds: 267.98 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 1534 (time 1734259599.34 total 10.00 last 153.33)
 - open/close 3142 (time 1734259609.34 total 20.01 last 160.71)
 - open/close 4857 (time 1734259619.35 total 30.01 last 171.45)
 - open/close 6296 (time 1734259629.35 total 40.01 last 143.88)
 - open/close 7882 (time 1734259639.35 total 50.02 last 158.52)
 - open/close 9425 (time 1734259649.36 total 60.02 last 154.24)
 - open/close 10000 (time 1734259652.86 total 63.53 last 164.07)
 - open/close 11524 (time 1734259662.86 total 73.53 last 152.34)
 - open/close 13082 (time 1734259672.86 total 83.53 last 155.78)
 - open/close 14596 (time 1734259682.87 total 93.54 last 151.34)
 - open/close 16246 (time 1734259692.87 total 103.54 last 164.99)
 - open/close 17765 (time 1734259702.87 total 113.54 last 151.86)
 - open/close 19303 (time 1734259712.88 total 123.55 last 153.72)
 - open/close 20000 (time 1734259718.13 total 128.80 last 132.73)
 - open/close 21506 (time 1734259728.13 total 138.80 last 150.56)
 - open/close 22997 (time 1734259738.14 total 148.81 last 148.98)
 - open/close 24675 (time 1734259748.14 total 158.81 last 167.74)
 - open/close 26275 (time 1734259758.15 total 168.82 last 159.91)
 - open/close 27827 (time 1734259768.16 total 178.82 last 155.07)
 - open/close 29600 (time 1734259778.16 total 188.83 last 177.28)
 - open/close 30000 (time 1734259780.21 total 190.88 last 194.70)
 - open/close 31851 (time 1734259790.22 total 200.88 last 185.02)
 - open/close 33627 (time 1734259800.22 total 210.89 last 177.53)
 - open/close 35373 (time 1734259810.22 total 220.89 last 174.55)
 - open/close 37191 (time 1734259820.23 total 230.89 last 181.78)
 - open/close 38841 (time 1734259830.23 total 240.89 last 165.00)
 - open/close 40000 (time 1734259838.13 total 248.80 last 146.58)
 - open/close 41468 (time 1734259848.14 total 258.80 last 146.73)
 - open/close 43086 (time 1734259858.14 total 268.81 last 161.78)
 - open/close 44529 (time 1734259868.14 total 278.81 last 144.24)
 - open/close 45972 (time 1734259878.15 total 288.81 last 144.24)
 - open/close 47576 (time 1734259888.15 total 298.82 last 160.38)
 - open/close 49192 (time 1734259898.15 total 308.82 last 161.51)
 - open/close 50000 (time 1734259903.22 total 313.88 last 159.62)
 - open/close 51772 (time 1734259913.22 total 323.89 last 177.12)
 - open/close 53628 (time 1734259923.22 total 333.89 last 185.57)
 - open/close 55409 (time 1734259933.22 total 343.89 last 178.05)
 - open/close 57243 (time 1734259943.23 total 353.89 last 183.35)
 - open/close 59102 (time 1734259953.23 total 363.90 last 185.86)
 - open/close 60000 (time 1734259957.85 total 368.52 last 194.26)
 - open/close 61849 (time 1734259967.85 total 378.52 last 184.84)
 - open/close 63673 (time 1734259977.86 total 388.53 last 182.33)
total: 64768 open/close in 394.50 seconds: 164.18 ops/second
 - /unlink 3166 (time 1734259995.05 total 10.00 last 316.58)
 - /unlink 6456 (time 1734260005.05 total 20.00 last 328.95)
 - /unlink 9684 (time 1734260015.05 total 30.00 last 322.77)
 - /unlink 12889 (time 1734260025.05 total 40.01 last 320.43)
 - /unlink 16196 (time 1734260035.05 total 50.01 last 330.62)
 - /unlink 19352 (time 1734260045.06 total 60.01 last 315.53)
 - /unlink 22607 (time 1734260055.06 total 70.01 last 325.48)
 - /unlink 25572 (time 1734260065.06 total 80.01 last 296.46)
 - /unlink 28235 (time 1734260075.06 total 90.02 last 266.17)
 - /unlink 30000 (time 1734260081.42 total 96.37 last 277.60)
 - /unlink 33020 (time 1734260091.42 total 106.38 last 301.90)
 - /unlink 35989 (time 1734260101.43 total 116.38 last 296.81)
 - /unlink 39022 (time 1734260111.43 total 126.38 last 303.29)
 - /unlink 40000 (time 1734260114.48 total 129.43 last 320.88)
 - /unlink 43006 (time 1734260124.48 total 139.43 last 300.54)
 - /unlink 46249 (time 1734260134.48 total 149.43 last 324.27)
 - /unlink 49639 (time 1734260144.48 total 159.44 last 338.89)
 - /unlink 53036 (time 1734260154.48 total 169.44 last 339.67)
 - /unlink 56838 (time 1734260164.48 total 179.44 last 380.14)
 - /unlink 60000 (time 1734260173.37 total 188.33 last 355.72)
 - /unlink 64134 (time 1734260183.37 total 198.33 last 413.36)
total: 64768 /unlink in 199.82 seconds: 324.13 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 2713 (time 1734260214.58 total 10.00 last 271.22)
 - open/close 5371 (time 1734260224.58 total 20.00 last 265.74)
 - open/close 8227 (time 1734260234.58 total 30.01 last 285.50)
 - open/close 10000 (time 1734260241.85 total 37.28 last 243.77)
 - open/close 12396 (time 1734260251.86 total 47.29 last 239.53)
 - open/close 14871 (time 1734260261.86 total 57.29 last 247.48)
 - open/close 17389 (time 1734260271.86 total 67.29 last 251.79)
 - open/close 20000 (time 1734260281.38 total 76.81 last 274.23)
 - open/close 22568 (time 1734260291.38 total 86.81 last 256.73)
 - open/close 25110 (time 1734260301.39 total 96.82 last 254.06)
 - open/close 27328 (time 1734260311.39 total 106.82 last 221.73)
 - open/close 29376 (time 1734260321.39 total 116.82 last 204.78)
 - open/close 30000 (time 1734260324.73 total 120.16 last 186.88)
 - open/close 32273 (time 1734260334.73 total 130.16 last 227.29)
 - open/close 34274 (time 1734260344.73 total 140.16 last 200.04)
 - open/close 36594 (time 1734260354.73 total 150.16 last 231.99)
 - open/close 39423 (time 1734260364.74 total 160.17 last 282.81)
 - open/close 40000 (time 1734260366.96 total 162.39 last 259.52)
 - open/close 42454 (time 1734260376.96 total 172.39 last 245.33)
 - open/close 44884 (time 1734260386.97 total 182.40 last 242.93)
 - open/close 47503 (time 1734260396.97 total 192.40 last 261.84)
 - open/close 50000 (time 1734260405.68 total 201.10 last 286.80)
 - open/close 52531 (time 1734260415.68 total 211.10 last 253.07)
 - open/close 54814 (time 1734260425.68 total 221.11 last 228.27)
 - open/close 57174 (time 1734260435.68 total 231.11 last 235.96)
 - open/close 59310 (time 1734260445.68 total 241.11 last 213.54)
 - open/close 60000 (time 1734260449.36 total 244.79 last 187.60)
 - open/close 62182 (time 1734260459.36 total 254.79 last 218.14)
 - open/close 64361 (time 1734260469.36 total 264.79 last 217.89)
total: 64768 open/close in 266.96 seconds: 242.61 ops/second
 - /unlink 3482 (time 1734260482.73 total 10.00 last 348.12)
 - /unlink 6951 (time 1734260492.73 total 20.00 last 346.83)
 - /unlink 10000 (time 1734260501.46 total 28.73 last 349.27)
 - /unlink 13957 (time 1734260511.46 total 38.73 last 395.68)
 - /unlink 18357 (time 1734260521.47 total 48.74 last 439.95)
 - /unlink 20000 (time 1734260524.96 total 52.23 last 469.99)
 - /unlink 24502 (time 1734260534.96 total 62.23 last 450.08)
 - /unlink 28638 (time 1734260544.96 total 72.24 last 413.55)
 - /unlink 30000 (time 1734260548.18 total 75.45 last 423.70)
 - /unlink 34067 (time 1734260558.18 total 85.45 last 406.66)
 - /unlink 38963 (time 1734260568.18 total 95.45 last 489.55)
 - /unlink 40000 (time 1734260570.23 total 97.50 last 506.26)
 - /unlink 45199 (time 1734260580.23 total 107.50 last 519.80)
 - /unlink 50000 (time 1734260589.99 total 117.26 last 491.82)
 - /unlink 54290 (time 1734260599.99 total 127.27 last 428.94)
 - /unlink 59728 (time 1734260610.00 total 137.27 last 543.75)
total: 64768 /unlink in 145.81 seconds: 444.21 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 2511 (time 1734260650.32 total 10.00 last 251.01)
 - open/close 4988 (time 1734260660.32 total 20.01 last 247.59)
 - open/close 7508 (time 1734260670.32 total 30.01 last 252.00)
 - open/close 10000 (time 1734260679.99 total 39.67 last 257.87)
 - open/close 12863 (time 1734260689.99 total 49.67 last 286.27)
 - open/close 15539 (time 1734260699.99 total 59.68 last 267.54)
 - open/close 18214 (time 1734260709.99 total 69.68 last 267.46)
 - open/close 20000 (time 1734260716.59 total 76.28 last 270.62)
 - open/close 22701 (time 1734260726.59 total 86.28 last 270.08)
 - open/close 25484 (time 1734260736.59 total 96.28 last 278.26)
 - open/close 28383 (time 1734260746.60 total 106.28 last 289.86)
 - open/close 30000 (time 1734260751.98 total 111.67 last 300.18)
 - open/close 33595 (time 1734260761.99 total 121.67 last 359.38)
 - open/close 37558 (time 1734260771.99 total 131.67 last 396.24)
 - open/close 40000 (time 1734260778.90 total 138.59 last 352.99)
 - open/close 44018 (time 1734260788.91 total 148.59 last 401.76)
 - open/close 48698 (time 1734260798.91 total 158.59 last 467.98)
 - open/close 50000 (time 1734260801.34 total 161.02 last 535.80)
 - open/close 55646 (time 1734260811.34 total 171.02 last 564.59)
 - open/close 60000 (time 1734260819.97 total 179.65 last 504.47)
 - open/close 64751 (time 1734260829.97 total 189.65 last 475.07)
total: 64768 open/close in 189.69 seconds: 341.44 ops/second
 - /unlink 8479 (time 1734260840.91 total 10.00 last 847.90)
 - /unlink 16371 (time 1734260850.91 total 20.00 last 789.13)
 - /unlink 20000 (time 1734260855.37 total 24.45 last 815.08)
 - /unlink 27098 (time 1734260865.37 total 34.45 last 709.74)
 - /unlink 30000 (time 1734260869.45 total 38.54 last 710.80)
 - /unlink 36903 (time 1734260879.45 total 48.54 last 690.26)
 - /unlink 40000 (time 1734260883.93 total 53.02 last 691.40)
 - /unlink 47869 (time 1734260893.93 total 63.02 last 786.84)
 - /unlink 50000 (time 1734260896.41 total 65.50 last 858.78)
 - /unlink 59562 (time 1734260906.41 total 75.50 last 956.12)
total: 64768 /unlink in 81.09 seconds: 798.74 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 5090 (time 1734260945.44 total 10.00 last 508.98)
 - open/close 10000 (time 1734260955.09 total 19.65 last 508.94)
 - open/close 15588 (time 1734260965.09 total 29.65 last 558.72)
 - open/close 20000 (time 1734260973.23 total 37.78 last 542.42)
 - open/close 25596 (time 1734260983.23 total 47.78 last 559.60)
 - open/close 30000 (time 1734260991.98 total 56.54 last 503.02)
 - open/close 35132 (time 1734261001.98 total 66.54 last 513.12)
 - open/close 40000 (time 1734261011.75 total 76.30 last 498.52)
 - open/close 45159 (time 1734261021.75 total 86.31 last 515.82)
 - open/close 49893 (time 1734261031.75 total 96.31 last 473.33)
 - open/close 54495 (time 1734261041.75 total 106.31 last 460.15)
 - open/close 59459 (time 1734261051.75 total 116.31 last 496.40)
 - open/close 64057 (time 1734261061.75 total 126.31 last 459.76)
total: 64768 open/close in 127.88 seconds: 506.47 ops/second
 - /unlink 7868 (time 1734261074.25 total 10.00 last 786.76)
 - /unlink 10000 (time 1734261076.91 total 12.67 last 799.40)
 - /unlink 18396 (time 1734261086.91 total 22.67 last 839.57)
 - /unlink 26851 (time 1734261096.92 total 32.67 last 845.41)
 - /unlink 30000 (time 1734261100.67 total 36.43 last 838.00)
 - /unlink 38317 (time 1734261110.67 total 46.43 last 831.64)
 - /unlink 40000 (time 1734261112.68 total 48.43 last 839.83)
 - /unlink 47991 (time 1734261122.68 total 58.43 last 799.04)
 - /unlink 50000 (time 1734261125.40 total 61.15 last 737.92)
 - /unlink 58190 (time 1734261135.40 total 71.16 last 818.91)
 - /unlink 60000 (time 1734261137.45 total 73.21 last 882.64)
total: 64768 /unlink in 78.64 seconds: 823.58 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
fail_loc=0
fail_val=0
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
PASS 106 (2388s)

== conf-sanity-1 test 109a: test lctl clear_conf fsname == 06:13:02 (1734261182)

 SKIP: conf-sanity-1 test_109a LU-8727: no implementation for ZFS
SKIP 109a (1s)

== conf-sanity-1 test 109b: test lctl clear_conf one config ========================================================== 06:13:03 (1734261183)

 SKIP: conf-sanity-1 test_109b LU-8727: no implementation for ZFS
SKIP 109b (0s)

== conf-sanity-1 test 116: big size MDT support ========== 06:13:04 (1734261184)

 SKIP: conf-sanity-1 test_116 ldiskfs only test
SKIP 116 (1s)

== conf-sanity-1 test 122a: Check OST sequence update ==== 06:13:04 (1734261184)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
fail_loc=0x00001e0
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
fail_loc=0
total: 1000 open/close in 2.20 seconds: 454.11 ops/second
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
PASS 122a (79s)

== conf-sanity-1 test 123ah: del_ost cancels config log entries correctly ========================================================== 06:14:23 (1734261263)
Checking servers environments
Checking clients oleg136-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
Starting client oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
Started clients oleg136-client.virtnet: 
192.168.201.136@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800a9d34800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800a9d34800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 7s: want 'procname_uid' got 'procname_uid'
disable quota as required
del_ost: dry run for target lustre-OST0000
config_log: lustre-MDT0001
[DRY RUN] cancel catalog 'lustre-MDT0001:32':"- { index: 32, event: add_osc, device: lustre-MDT0001-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-MDT0001:31':"- { index: 31, event: setup, device: lustre-OST0000-osc-MDT0001, UUID: lustre-OST0000_UUID, node: 192.168.201.136@tcp }"
[DRY RUN] cancel catalog 'lustre-MDT0001:30':"- { index: 30, event: attach, device: lustre-OST0000-osc-MDT0001, type: osc, UUID: lustre-MDT0001-mdtlov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-MDT0000
[DRY RUN] cancel catalog 'lustre-MDT0000:29':"- { index: 29, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-MDT0000:28':"- { index: 28, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.201.136@tcp }"
[DRY RUN] cancel catalog 'lustre-MDT0000:27':"- { index: 27, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-client
[DRY RUN] cancel catalog 'lustre-client:34':"- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-client:33':"- { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.201.136@tcp }"
[DRY RUN] cancel catalog 'lustre-client:32':"- { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-MDT0001
cancel catalog lustre-MDT0001 log_idx 32: done
cancel catalog lustre-MDT0001 log_idx 31: done
cancel catalog lustre-MDT0001 log_idx 30: done
del_ost: cancelled 3 catalog entries
config_log: lustre-MDT0000
cancel catalog lustre-MDT0000 log_idx 29: done
cancel catalog lustre-MDT0000 log_idx 28: done
cancel catalog lustre-MDT0000 log_idx 27: done
del_ost: cancelled 3 catalog entries
config_log: lustre-client
cancel catalog lustre-client log_idx 34: done
cancel catalog lustre-client log_idx 33: done
cancel catalog lustre-client log_idx 32: done
del_ost: cancelled 3 catalog entries
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
oleg136-server: rmmod: ERROR: Module lustre is in use
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
fail_loc=0
PASS 123ah (112s)

== conf-sanity-1 test 132: hsm_actions processed after failover ========================================================== 06:16:15 (1734261375)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mdt.hsm_control=enabled
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:mdt.hsm_control=enabled
  lustre:version=1
  lustre:flags=69
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
PASS 132 (26s)

== conf-sanity-1 test 133: stripe QOS: free space balance in a pool ========================================================== 06:16:41 (1734261401)

 SKIP: conf-sanity-1 test_133 needs >= 4 OSTs
SKIP 133 (1s)

== conf-sanity-1 test 135: check the behavior when changelog is wrapped around ========================================================== 06:16:42 (1734261402)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
oleg136-client: fail_loc=0x1312
oleg136-client: fail_val=5
oleg136-server: fail_loc=0x1312
oleg136-server: fail_val=5
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity-1
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
total: 4500 open/close in 9.84 seconds: 457.29 ops/second
total: 4500 /unlink in 5.40 seconds: 833.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 4495 (time 1734261463.58 total 10.00 last 449.42)
total: 4500 open/close in 10.01 seconds: 449.52 ops/second
total: 4500 /unlink in 5.32 seconds: 846.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 4418 (time 1734261480.90 total 10.00 last 441.74)
total: 4500 open/close in 10.18 seconds: 442.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
total: 4500 /unlink in 5.27 seconds: 853.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 4443 (time 1734261498.07 total 10.00 last 444.28)
total: 4500 open/close in 10.12 seconds: 444.55 ops/second
total: 4500 /unlink in 5.59 seconds: 805.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
total: 4500 open/close in 9.98 seconds: 451.09 ops/second
total: 4500 /unlink in 5.30 seconds: 848.80 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 10973: 13195 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg136-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
PASS 135 (149s)

== conf-sanity-1 test 136: don't panic with bad obdecho setup ========================================================== 06:19:11 (1734261551)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
oleg136-server: oleg136-server.virtnet: executing load_module obdecho/obdecho
oleg136-server: error: setup: Invalid argument
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22
oleg136-server: error: test_mkdir: No such file or directory
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:)
Stopping client oleg136-client.virtnet /mnt/lustre opts:
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 136 (117s)

== conf-sanity-1 test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:21:08 (1734261668)

 SKIP: conf-sanity-1 test_137 needs >= 3 MDTs
SKIP 137 (0s)

== conf-sanity-1 test 140: remove_updatelog script actions ========================================================== 06:21:09 (1734261669)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
striped dir -i0 -c2 -H all_char /mnt/lustre/d140.conf-sanity-1
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
Dry run was requested, no changes will be applied
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 0 1
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
Processing MDT1 llog catalog [0x240000400:0x1:0x0] ...
remove_updatelog: /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] is too small.
> /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0]
Dry run was requested, no changes will be applied
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 1 0
Processing MDT1 llog catalog [0x240000400:0x1:0x0] ...
remove_updatelog: /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] is too small.
> /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0]
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 0
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:)
Stopping client oleg136-client.virtnet /mnt/lustre opts:
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
PASS 140 (230s)

== conf-sanity-1 test 151a: damaged local config doesn't prevent mounting ========================================================== 06:24:59 (1734261899)

 SKIP: conf-sanity-1 test_151a ldiskfs only test
SKIP 151a (1s)

== conf-sanity-1 test 151b: -ENOSPC doesn't affect mount ========================================================== 06:25:00 (1734261900)

 SKIP: conf-sanity-1 test_151b ldiskfs only test
SKIP 151b (1s)

== conf-sanity-1 test 153a: bypass invalid NIDs quickly == 06:25:01 (1734261901)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
waiting for mount ...
          "192.168.201.136@tcp": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 5 }
          "192.168.252.112@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
          "10.252.252.113@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
          "192.168.201.136@tcp": { connects: 0, replied: 0, uptodate: unknown, sec_ago: never }
setup single mount lustre success
umount lustre on /mnt/lustre.....
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
PASS 153a (196s)

== conf-sanity-1 test 153b: added IPv6 NID support ======= 06:28:17 (1734262097)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
Checking servers environments
Checking clients oleg136-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
Starting client oleg136-client.virtnet:  -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre
Started clients oleg136-client.virtnet: 
192.168.201.136@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b5821800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b5821800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 5s: want 'procname_uid' got 'procname_uid'
disable quota as required
PASS 153b (107s)

== conf-sanity-1 test 153c: don't stuck on unreached NID ========================================================== 06:30:04 (1734262204)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
start mds service on oleg136-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg136-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg136-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1
Started lustre-OST0000
oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
          "192.168.252.112@tcp": { connects: 3, replied: 0, uptodate: discovering, sec_ago: 0 }
          "192.168.252.113@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 10 }
          "192.168.252.115@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 5 }
connection attempts: 7
Waiting for mount to fail
mount.lustre: mount 192.168.252.112@tcp:192.168.252.113@tcp:192.168.252.115@tcp:/lustre at /mnt/lustre failed: Input/output error
Is the MGS running?
umount lustre on /mnt/lustre.....
stop ost1 service on oleg136-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server
stop mds service on oleg136-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server
unloading modules via unload_modules_local on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
PASS 153c (359s)

== conf-sanity-1 test 154: expand .. on rename after MDT backup restore ========================================================== 06:36:03 (1734262563)

 SKIP: conf-sanity-1 test_154 ldiskfs only test
SKIP 154 (1s)
Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2
oleg136-server: oleg136-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg136-server'
oleg136-server: oleg136-server.virtnet: executing load_modules_local
oleg136-server: Loading modules from /home/green/git/lustre-release/lustre
oleg136-server: detected 4 online CPUs by sysfs
oleg136-server: Force libcfs to create 2 CPU partitions
oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
== conf-sanity-1 test complete, duration 7477 sec ======== 06:36:31 (1734262591)
=== conf-sanity-1: start cleanup 06:36:32 (1734262592) ===
=== conf-sanity-1: finish cleanup 06:36:33 (1734262593) ===