-----============= acceptance-small: conf-sanity-1 ============----- Sun Dec 15 04:31:41 EST 2024
mgs: CentOS Linux release 7.9.2009 (Core)
MGS_OS_VERSION_ID=7
MGS_OS_ID=centos
MGS_OS_VERSION_CODE=117440512
MGS_OS_ID_LIKE=rhel fedora centos
mds1: CentOS Linux release 7.9.2009 (Core)
MDS1_OS_ID_LIKE=rhel fedora centos
MDS1_OS_ID=centos
MDS1_OS_VERSION_ID=7
MDS1_OS_VERSION_CODE=117440512
ost1: CentOS Linux release 7.9.2009 (Core)
OST1_OS_VERSION_CODE=117440512
OST1_OS_VERSION_ID=7
OST1_OS_ID_LIKE=rhel fedora centos
OST1_OS_ID=centos
client: CentOS Linux release 7.9.2009 (Core)
CLIENT_OS_ID=centos
CLIENT_OS_ID_LIKE=rhel fedora centos
CLIENT_OS_VERSION_ID=7
CLIENT_OS_VERSION_CODE=117440512
oleg122-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity-1.*ex: No such file or directory
excepting tests: 
skipping tests SLOW=no: 106
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg122-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server

== conf-sanity-1 test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 04:32:52 (1734255172)

 SKIP: conf-sanity-1 test_17 ldiskfs only test
SKIP 17 (1s)

== conf-sanity-1 test 18: check mkfs creates large journals ========================================================== 04:32:53 (1734255173)

 SKIP: conf-sanity-1 test_18 ldiskfs only test
SKIP 18 (1s)

== conf-sanity-1 test 21c: start mds between two osts, stop mds last ========================================================== 04:32:54 (1734255174)
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost2 service on oleg122-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop ost2 service on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 21c (114s)

== conf-sanity-1 test 29: permanently remove an OST ====== 04:34:48 (1734255288)
start ost2 service on oleg122-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Setting lustre-OST0001.osc.active from 1 to 0
Waiting 90s for '0'
Updated after 2s: want '0' got '0'
Live client success: got lustre-OST0001_UUID	FULL	DEACTIVATED
osc.lustre-OST0001-osc-MDT0000.active updated after 0 sec (got 0)
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
New client success: got 'lustre-OST0001_UUID	NEW	DEACTIVATED'
Setting lustre-OST0001.osc.active from 0 to 1
Waiting 90s for '1'
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 29 (95s)

== conf-sanity-1 test 35a: Reconnect to the last active server first ========================================================== 04:36:23 (1734255383)
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
debug=ha
Set up a fake failnode for the MDS
Wait for RECONNECT_INTERVAL seconds (10s)
conf-sanity.sh test_35a 2024-12-15 4h36m50s
Stopping the MDT: lustre-MDT0000
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
Restarting the MDT: lustre-MDT0000
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
Wait for df (23775) ... 
done
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout
Debug log: 70 lines, 70 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 35a (92s)

== conf-sanity-1 test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:37:55 (1734255475)
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
debug=ha
conf-sanity.sh test_35b 2024-12-15 4h38m11s
Set up a fake failnode for the MDS
at_max=0
at_max=0
Injecting EBUSY on MDS
fail_loc=0x80000136
mdc.lustre-MDT0000-mdc-ffff8800aae98000.stats=clear
Creating a test file and stat it
  File: '/mnt/lustre/d35b.conf-sanity-1/f35b.conf-sanity-1'
  Size: 0         	Blocks: 1          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d	Inode: 144115238826934274  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:38:54.000000000 -0500
Modify: 2024-12-15 04:38:54.000000000 -0500
Change: 2024-12-15 04:38:54.000000000 -0500
 Birth: -
Stop injecting EBUSY on MDS
fail_loc=0
done
at_max=600
at_max=600
Debug log: 48 lines, 48 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 35b (119s)

== conf-sanity-1 test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:39:54 (1734255594)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov -o nosvc -n  lustre-mdt1/mdt1 /mnt/lustre-mds1
nomtab: 1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
Start lustre-mdt1/mdt1 without service
Commit the device label on lustre-mdt1/mdt1
Started lustre:MDT0000
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg122-server
Starting mds1: -o localrecov -o nomgs,force  lustre-mdt1/mdt1 /mnt/lustre-mds1
force: 1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
blah blah
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 41b (64s)

== conf-sanity-1 test 46a: handle ost additional - wide striped file ========================================================== 04:40:58 (1734255658)
Testing with 2 OSTs
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
oleg122-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a5de000.ost_server_uuid 50
osc.lustre-OST0001-osc-ffff88012a5de000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre2.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre2
/mnt/lustre2
stripe_count:  -1 stripe_size:   4194304 pattern:       raid0 stripe_offset: -1

/mnt/lustre2/f41b.conf-sanity-1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	             2	          0x2	   0x240000400

/mnt/lustre2/widestripe
lmm_stripe_count:  2
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 1
	obdidx		 objid		 objid		 group
	     1	             2	          0x2	   0x280000400
	     0	            34	         0x22	   0x240000400

  File: '/mnt/lustre/widestripe'
  Size: 3         	Blocks: 2          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115238826934273  Links: 1
Access: (0674/-rw-rwxr--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:41:27.000000000 -0500
Modify: 2024-12-15 04:41:27.000000000 -0500
Change: 2024-12-15 04:41:28.000000000 -0500
 Birth: -
umount lustre on /mnt/lustre2.....
Stopping client oleg122-client.virtnet /mnt/lustre2 (opts:)
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
stop ost1 service on oleg122-server
stop mds service on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 46a (130s)

== conf-sanity-1 test 47a: server restart does not lose lru_resize ========================================================== 04:43:08 (1734255788)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ac471800.lru_size=100
ldlm.namespaces.lustre-OST0000-osc-ffff8800ac471800.lru_size=100
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ac471800.lru_size=100
Failing ost1 on oleg122-server
Stopping /mnt/lustre-ost1 (opts:) on oleg122-server
04:43:44 (1734255824) shut down
facet: ost1 facet_host: oleg122-server facet_failover_host: oleg122-server
Failover ost1 to oleg122-server
mount facets: ost1
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
04:43:58 (1734255838) targets are mounted
04:43:58 (1734255838) facet_failover done
Failing mds1 on oleg122-server
Stopping /mnt/lustre-mds1 (opts:) on oleg122-server
04:43:59 (1734255839) shut down
facet: mds1 facet_host: oleg122-server facet_failover_host: oleg122-server
Failover mds1 to oleg122-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
04:44:12 (1734255852) targets are mounted
04:44:12 (1734255852) facet_failover done
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ac471800.lru_size=100
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
PASS 47a (83s)

== conf-sanity-1 test 47b: client restart does not lose lru_resize ========================================================== 04:44:31 (1734255871)
Failing mds1 on oleg122-server
04:44:32 (1734255872) shut down
facet: mds1 facet_host: oleg122-server facet_failover_host: oleg122-server
Failover mds1 to oleg122-server
mount facets: mds1
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
04:44:57 (1734255897) targets are mounted
04:44:57 (1734255897) facet_failover done
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 95
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 95
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
ln: failed to create symbolic link '/home/green/git/lustre-release/lustre/utils/lctl': File exists

 SKIP: conf-sanity-1 test_47b Can not create symlink /usr/sbin/lctl
SKIP 47b (75s)

== conf-sanity-1 test 48: too many acls on file ========== 04:45:46 (1734255946)
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 17
Start of lustre-mdt1/mdt1 on mds1 failed 17
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg122-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of lustre-ost1/ost1 on ost1 failed 17
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
mount.lustre: according to /etc/mtab 192.168.201.122@tcp:/lustre is already mounted on /mnt/lustre
setup single mount lustre success
debug=0
debug=0
/mnt/lustre
stripe_count:  -1 stripe_size:   4194304 pattern:       raid0 stripe_offset: -1

/mnt/lustre/widestripe
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	             4	          0x4	   0x240000400

It is expected to hold at least 4500 ACL entries
  File: '/mnt/lustre/widestripe'
  Size: 3         	Blocks: 1          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115205272502275  Links: 1
Access: (0664/-rw-rw-r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:45:51.000000000 -0500
Modify: 2024-12-15 04:45:51.000000000 -0500
Change: 2024-12-15 04:47:20.000000000 -0500
 Birth: -
getfacl: Removing leading '/' from absolute path names
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg122-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 48 (162s)

== conf-sanity-1 test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:48:28 (1734256108)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
check ldlm_timout...
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 49a (34s)

== conf-sanity-1 test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:49:02 (1734256142)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
PASS 49b (50s)

== conf-sanity-1 test 50c: lazystatfs one server down ==== 04:49:52 (1734256192)
start mds service on oleg122-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg122-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800aa56f000.lazystatfs=1
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
oleg122-server: oleg122-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7006
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-OST0001_UUID       382976        3072      377856   1% /mnt/lustre[OST:1]

filesystem_summary:       382976        3072      377856   1% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (5722) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 50c (98s)

== conf-sanity-1 test 50d: lazystatfs client/server conn race ========================================================== 04:51:30 (1734256290)
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg122-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800ae97f000.lazystatfs=1
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7006
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-OST0001_UUID       382976        3072      377856   1% /mnt/lustre[OST:1]

filesystem_summary:       382976        3072      377856   1% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (10670) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 50d (88s)

== conf-sanity-1 test 50f: normal statfs one server in down ========================================================== 04:52:58 (1734256378)
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
start ost2 service on oleg122-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
oleg122-server: oleg122-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800aebe8000.lazystatfs=0
multiop /mnt/lustre v_f
TMPPIPE=/tmp/multiop_open_wait_pipe.7006
start ost2 service on oleg122-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0001
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 50f (115s)

== conf-sanity-1 test 50g: deactivated OST should not cause panic ========================================================== 04:54:53 (1734256493)
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
start ost2 service on oleg122-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec
oleg122-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800aebea800.ost_server_uuid 50
osc.lustre-OST0001-osc-ffff8800aebea800.ost_server_uuid in FULL state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
Filesystem                  1K-blocks  Used Available Use% Mounted on
192.168.201.122@tcp:/lustre    382976  3072    377856   1% /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 50g (75s)

== conf-sanity-1 test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:56:08 (1734256568)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
fail_loc=0x142
start ost2 service on oleg122-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec
stop ost2 service on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 51 (126s)

== conf-sanity-1 test 54a: test llverdev and partial verify of device ========================================================== 04:58:15 (1734256695)

 SKIP: conf-sanity-1 test_54a ldiskfs only test
SKIP 54a (0s)

== conf-sanity-1 test 55: check lov_objid size =========== 04:58:16 (1734256696)

 SKIP: conf-sanity-1 test_55 ldiskfs only test
SKIP 55 (0s)

== conf-sanity-1 test 56a: check big OST indexes and out-of-index-order start ========================================================== 04:58:17 (1734256697)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2

   Permanent disk data:
Target:     lustre:OST2710
Index:      10000
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.122@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=10000
  lustre:fsname=lustre
  lustre:svname=lustre:OST2710

   Permanent disk data:
Target:     lustre:OST03e8
Index:      1000
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost2 /dev/vdf
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost2/ost2
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost2/ost2 properties
  lustre:mgsnode=192.168.201.122@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=1000
  lustre:fsname=lustre
  lustre:svname=lustre:OST03e8
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST2710-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST2710
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg122-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST03e8-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST03e8
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
ok
OBDS:
1000: lustre-OST03e8_UUID ACTIVE
10000: lustre-OST2710_UUID ACTIVE
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.00427593 s, 958 kB/s
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:)
Stopping client oleg122-client.virtnet /mnt/lustre opts:
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 56a (118s)

== conf-sanity-1 test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:00:15 (1734256815)

 SKIP: conf-sanity-1 test_56b needs >= 3 MDTs
SKIP 56b (1s)

== conf-sanity-1 test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:00:16 (1734256816)
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 autodegrade=on failover.node=192.168.201.122@tcp autodegrade=on
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.122@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:failover.node=192.168.201.122@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg122-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Cannot assign requested address
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 99
oleg122-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg122-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
Start of lustre-ost1/ost1 on ost1 failed 99
umount lustre on /mnt/lustre.....
stop ost1 service on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
PASS 57a (75s)

== conf-sanity-1 test 57b: initial registration from servicenode should not fail ========================================================== 05:01:31 (1734256891)
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.122@tcp

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x1042
              (OST update no_primnode )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.122@tcp failover.node=192.168.201.122@tcp autodegrade=on
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.201.122@tcp
  lustre:failover.node=192.168.201.122@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=4162
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
PASS 57b (95s)

== conf-sanity-1 test 59: writeconf mount option ========= 05:03:06 (1734256986)
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
original ost count: 1 (expect > 0)
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
after mdt writeconf count: 0 (expect 0)
OST start without writeconf should fail:
oleg122-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: No such file or directory
oleg122-server: Is the MGS specification correct?
oleg122-server: Is the filesystem name correct?
oleg122-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg122-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
OST start with writeconf should succeed:
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
after ost writeconf count: 1 (expect 1)
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
after ost2 writeconf count: 2 (expect 2)
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 59 (116s)

== conf-sanity-1 test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 05:05:03 (1734257103)

 SKIP: conf-sanity-1 test_60a ldiskfs only test
SKIP 60a (1s)

== conf-sanity-1 test 62: start with disabled journal ==== 05:05:04 (1734257104)

 SKIP: conf-sanity-1 test_62 ldiskfs only test
SKIP 62 (1s)

== conf-sanity-1 test 64: check lfs df --lazy ============ 05:05:05 (1734257105)
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg122-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
stop ost2 service on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
/home/green/git/lustre-release/lustre/utils/lfs df
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID       382976        3072      377856   1% /mnt/lustre[OST:0]

filesystem_summary:       382976        3072      377856   1% /mnt/lustre

umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg122-server: 
oleg122-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg122-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 64 (127s)

== conf-sanity-1 test 66: replace nids =================== 05:07:12 (1734257232)
client=34615898 MDS=34615898 OSS=34615898
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
Setting lustre-OST0000.osc.active from 1 to 0
Waiting 90s for '0'
Updated after 9s: want '0' got '0'
replace_nids should fail if MDS, OSTs and clients are UP
oleg122-server: error: replace_nids: Operation now in progress
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 115
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
replace_nids should fail if MDS and OSTs are UP
oleg122-server: error: replace_nids: Operation now in progress
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 115
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
replace_nids should fail if MDS is UP
oleg122-server: error: replace_nids: Operation now in progress
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 115
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
start mds service on oleg122-server
Starting mds1: -o localrecov -o nosvc  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
Start lustre-mdt1/mdt1 without service
Started lustre-MDT0000
command should accept two parameters
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
correct device name should be passed
oleg122-server: error: replace_nids: No such device or address
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 6
wrong nids list should not destroy the system
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
replace OST nid
command should accept two parameters
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
wrong nids list should not destroy the system
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
set NIDs with failover
replace MDS nid
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Setting lustre-OST0000.osc.active from 0 to 1
Waiting 90s for '1'
Updated after 3s: want '1' got '1'
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 66 (153s)

== conf-sanity-1 test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 05:09:45 (1734257385)

 SKIP: conf-sanity-1 test_70e needs >= 2 MDTs
SKIP 70e (1s)

== conf-sanity-1 test 75: The order of --index should be irrelevant ========================================================== 05:09:46 (1734257386)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1/mdt1
  xattr=sa
  dnodesize=auto
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=69
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.122@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /dev/vdc
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1/mdt1
  xattr=sa
  dnodesize=auto
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=69
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.122@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 75 (61s)

== conf-sanity-1 test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:10:47 (1734257447)

 SKIP: conf-sanity-1 test_78 ldiskfs only test
SKIP 78 (1s)

== conf-sanity-1 test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:10:48 (1734257448)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg122-server: 
oleg122-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg122-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 22
oleg122-server: 
oleg122-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg122-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 22
oleg122-server: 
oleg122-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg122-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 22
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 79 (60s)

== conf-sanity-1 test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 05:11:48 (1734257508)

 SKIP: conf-sanity-1 test_83 ldiskfs only test
SKIP 83 (1s)

== conf-sanity-1 test 86: Replacing mkfs.lustre -G option ========================================================== 05:11:49 (1734257509)

 SKIP: conf-sanity-1 test_86 LU-6442: no such mkfs params for ZFS OSTs
SKIP 86 (2s)

== conf-sanity-1 test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 05:11:51 (1734257511)

 SKIP: conf-sanity-1 test_87 ldiskfs only test
SKIP 87 (1s)

== conf-sanity-1 test 89: check tunefs --param and --erase-param{s} options ========================================================== 05:11:52 (1734257512)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:)
tunefs --param failover.node=192.0.2.254@tcp0
tunefs --param failover.node=192.0.2.255@tcp0
tunefs --erase-param failover.node
tunefs --erase-params
tunefs --param failover.node=192.0.2.254@tcp0 --erase-params
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 89 (49s)

== conf-sanity-1 test 93: register mulitple MDT at the same time ========================================================== 05:12:41 (1734257561)

 SKIP: conf-sanity-1 test_93 needs >= 3 MDTs
SKIP 93 (1s)

== conf-sanity-1 test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:12:42 (1734257562)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
lustre-MDT0000:
lustre-OST0000:
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
PASS 100 (50s)

== conf-sanity-1 test 104a: Make sure user defined options are reflected in mount ========================================================== 05:13:32 (1734257612)
mountfsopt: acl,user_xattr
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
Starting mds1: -o localrecov,noacl  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
setfacl: /mnt/lustre: Operation not supported
PASS 104a (47s)

== conf-sanity-1 test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:14:19 (1734257659)
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
oleg122-server: rmmod: ERROR: Module lustre is in use
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
touch: cannot touch '/tmp/d105.conf-sanity-1/f105.conf-sanity-1': Read-only file system
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
PASS 105 (96s)

== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:15:55 (1734257755)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        86842         329       86513   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID        16134         326       15808   3% /mnt/lustre[OST:0]

filesystem_summary:        16137         329       15808   3% /mnt/lustre

seq.cli-lustre-OST0000-super.width=0x1ffffff
fail_loc=0x1312
fail_val=5
 - open/close 2053 (time 1734257810.56 total 10.00 last 205.29)
 - open/close 4070 (time 1734257820.56 total 20.00 last 201.69)
 - open/close 6326 (time 1734257830.56 total 30.00 last 225.56)
 - open/close 8317 (time 1734257840.56 total 40.00 last 199.08)
 - open/close 10000 (time 1734257849.51 total 48.95 last 188.10)
 - open/close 11503 (time 1734257859.51 total 58.95 last 150.26)
 - open/close 13355 (time 1734257869.51 total 68.96 last 185.18)
 - open/close 15133 (time 1734257879.52 total 78.96 last 177.76)
 - open/close 17215 (time 1734257889.52 total 88.96 last 208.17)
 - open/close 19878 (time 1734257899.52 total 98.96 last 266.23)
 - open/close 22013 (time 1734257909.52 total 108.96 last 213.43)
 - open/close 23621 (time 1734257919.52 total 118.97 last 160.80)
 - open/close 25118 (time 1734257929.52 total 128.97 last 149.69)
 - open/close 26781 (time 1734257939.53 total 138.97 last 166.27)
 - open/close 28909 (time 1734257949.53 total 148.97 last 212.78)
 - open/close 30000 (time 1734257954.84 total 154.28 last 205.35)
 - open/close 31728 (time 1734257964.84 total 164.28 last 172.77)
 - open/close 33460 (time 1734257974.85 total 174.29 last 173.15)
 - open/close 34898 (time 1734257984.85 total 184.29 last 143.73)
 - open/close 36210 (time 1734257994.86 total 194.30 last 131.11)
 - open/close 37431 (time 1734258004.86 total 204.30 last 122.07)
 - open/close 38719 (time 1734258014.87 total 214.31 last 128.68)
 - open/close 40000 (time 1734258024.32 total 223.76 last 135.48)
 - open/close 41407 (time 1734258034.34 total 233.78 last 140.52)
 - open/close 42573 (time 1734258044.34 total 243.78 last 116.55)
 - open/close 43635 (time 1734258054.35 total 253.79 last 106.13)
 - open/close 44747 (time 1734258064.36 total 263.80 last 111.07)
 - open/close 45865 (time 1734258074.36 total 273.80 last 111.79)
 - open/close 46863 (time 1734258084.36 total 283.80 last 99.79)
 - open/close 48157 (time 1734258094.36 total 293.81 last 129.36)
 - open/close 49404 (time 1734258104.36 total 303.81 last 124.69)
 - open/close 50000 (time 1734258109.26 total 308.70 last 121.86)
 - open/close 51220 (time 1734258119.26 total 318.70 last 121.98)
 - open/close 52504 (time 1734258129.26 total 328.70 last 128.39)
 - open/close 53778 (time 1734258139.26 total 338.70 last 127.34)
 - open/close 55106 (time 1734258149.27 total 348.71 last 132.74)
 - open/close 56333 (time 1734258159.27 total 358.71 last 122.64)
 - open/close 57654 (time 1734258169.28 total 368.72 last 132.06)
 - open/close 59187 (time 1734258179.28 total 378.72 last 153.19)
 - open/close 60000 (time 1734258185.11 total 384.55 last 139.41)
 - open/close 61365 (time 1734258195.11 total 394.55 last 136.50)
 - open/close 62837 (time 1734258205.12 total 404.56 last 147.16)
 - open/close 64296 (time 1734258215.12 total 414.56 last 145.89)
total: 64768 open/close in 418.18 seconds: 154.88 ops/second
 - /unlink 1914 (time 1734258230.51 total 10.00 last 191.39)
 - /unlink 3779 (time 1734258240.51 total 20.00 last 186.44)
 - /unlink 5642 (time 1734258250.51 total 30.00 last 186.30)
 - /unlink 7428 (time 1734258260.52 total 40.01 last 178.52)
 - /unlink 9364 (time 1734258270.52 total 50.01 last 193.59)
 - /unlink 10000 (time 1734258274.13 total 53.62 last 176.21)
 - /unlink 11763 (time 1734258284.13 total 63.62 last 176.28)
 - /unlink 13390 (time 1734258294.13 total 73.62 last 162.63)
 - /unlink 15351 (time 1734258304.14 total 83.62 last 196.08)
 - /unlink 17209 (time 1734258314.14 total 93.63 last 185.77)
 - /unlink 18943 (time 1734258324.14 total 103.63 last 173.32)
 - /unlink 20000 (time 1734258329.84 total 109.33 last 185.53)
 - /unlink 21739 (time 1734258339.84 total 119.33 last 173.88)
 - /unlink 23737 (time 1734258349.84 total 129.33 last 199.76)
 - /unlink 25970 (time 1734258359.84 total 139.33 last 223.27)
 - /unlink 28261 (time 1734258369.84 total 149.33 last 229.07)
 - /unlink 30000 (time 1734258378.51 total 158.00 last 200.57)
 - /unlink 31953 (time 1734258388.51 total 168.00 last 195.29)
 - /unlink 33944 (time 1734258398.52 total 178.01 last 199.01)
 - /unlink 35940 (time 1734258408.53 total 188.01 last 199.48)
 - /unlink 37906 (time 1734258418.53 total 198.02 last 196.55)
 - /unlink 40000 (time 1734258428.31 total 207.80 last 213.96)
 - /unlink 41999 (time 1734258438.32 total 217.81 last 199.86)
 - /unlink 44286 (time 1734258448.32 total 227.81 last 228.64)
 - /unlink 46835 (time 1734258458.32 total 237.81 last 254.88)
 - /unlink 49372 (time 1734258468.33 total 247.82 last 253.56)
 - /unlink 50000 (time 1734258470.70 total 250.19 last 264.53)
 - /unlink 53086 (time 1734258480.70 total 260.19 last 308.49)
 - /unlink 56067 (time 1734258490.71 total 270.20 last 298.03)
 - /unlink 59452 (time 1734258500.71 total 280.20 last 338.48)
 - /unlink 62320 (time 1734258510.71 total 290.20 last 286.78)
total: 64768 /unlink in 298.97 seconds: 216.64 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 1297 (time 1734258548.29 total 10.00 last 129.64)
 - open/close 2566 (time 1734258558.29 total 20.01 last 126.84)
 - open/close 3872 (time 1734258568.30 total 30.01 last 130.54)
 - open/close 5182 (time 1734258578.30 total 40.01 last 130.99)
 - open/close 6451 (time 1734258588.30 total 50.02 last 126.84)
 - open/close 7698 (time 1734258598.32 total 60.04 last 124.46)
 - open/close 9100 (time 1734258608.32 total 70.04 last 140.18)
 - open/close 10000 (time 1734258615.44 total 77.16 last 126.47)
 - open/close 11253 (time 1734258625.44 total 87.16 last 125.28)
 - open/close 12562 (time 1734258635.45 total 97.16 last 130.82)
 - open/close 13798 (time 1734258645.45 total 107.17 last 123.57)
 - open/close 15120 (time 1734258655.46 total 117.17 last 132.15)
 - open/close 16491 (time 1734258665.46 total 127.17 last 137.07)
 - open/close 17960 (time 1734258675.46 total 137.18 last 146.84)
 - open/close 19308 (time 1734258685.47 total 147.18 last 134.74)
 - open/close 20000 (time 1734258689.90 total 151.61 last 156.24)
 - open/close 21510 (time 1734258699.90 total 161.61 last 150.96)
 - open/close 22933 (time 1734258709.91 total 171.62 last 142.19)
 - open/close 24193 (time 1734258719.91 total 181.63 last 125.93)
 - open/close 25363 (time 1734258729.92 total 191.64 last 116.91)
 - open/close 26628 (time 1734258739.93 total 201.64 last 126.39)
 - open/close 27796 (time 1734258749.93 total 211.65 last 116.74)
 - open/close 29048 (time 1734258759.94 total 221.65 last 125.17)
 - open/close 30000 (time 1734258767.71 total 229.42 last 122.54)
 - open/close 31316 (time 1734258777.71 total 239.42 last 131.59)
 - open/close 32674 (time 1734258787.71 total 249.42 last 135.78)
 - open/close 34224 (time 1734258797.71 total 259.42 last 154.97)
 - open/close 35715 (time 1734258807.71 total 269.43 last 149.02)
 - open/close 37053 (time 1734258817.72 total 279.43 last 133.79)
 - open/close 38691 (time 1734258827.72 total 289.43 last 163.80)
 - open/close 40000 (time 1734258835.99 total 297.70 last 158.28)
 - open/close 41546 (time 1734258845.99 total 307.70 last 154.53)
 - open/close 43007 (time 1734258855.99 total 317.71 last 146.09)
 - open/close 44324 (time 1734258865.99 total 327.71 last 131.69)
 - open/close 45793 (time 1734258875.99 total 337.71 last 146.88)
 - open/close 47220 (time 1734258885.99 total 347.71 last 142.69)
 - open/close 48637 (time 1734258895.99 total 357.71 last 141.69)
 - open/close 50000 (time 1734258905.43 total 367.15 last 144.37)
 - open/close 51423 (time 1734258915.44 total 377.15 last 142.26)
 - open/close 52924 (time 1734258925.44 total 387.15 last 150.08)
 - open/close 54602 (time 1734258935.44 total 397.16 last 167.74)
 - open/close 56312 (time 1734258945.45 total 407.16 last 170.92)
 - open/close 57992 (time 1734258955.45 total 417.16 last 167.99)
 - open/close 59509 (time 1734258965.45 total 427.17 last 151.59)
 - open/close 60000 (time 1734258969.34 total 431.06 last 126.30)
 - open/close 61362 (time 1734258979.35 total 441.06 last 136.16)
 - open/close 62708 (time 1734258989.35 total 451.07 last 134.53)
 - open/close 64120 (time 1734258999.35 total 461.07 last 141.14)
total: 64768 open/close in 465.57 seconds: 139.12 ops/second
 - /unlink 2603 (time 1734259015.43 total 10.00 last 260.26)
 - /unlink 5279 (time 1734259025.44 total 20.00 last 267.54)
 - /unlink 7793 (time 1734259035.44 total 30.01 last 251.35)
 - /unlink 10000 (time 1734259044.64 total 39.20 last 239.98)
 - /unlink 12437 (time 1734259054.64 total 49.21 last 243.64)
 - /unlink 14948 (time 1734259064.64 total 59.21 last 250.97)
 - /unlink 17190 (time 1734259074.64 total 69.21 last 224.19)
 - /unlink 19616 (time 1734259084.64 total 79.21 last 242.57)
 - /unlink 22240 (time 1734259094.65 total 89.21 last 262.38)
 - /unlink 24580 (time 1734259104.65 total 99.21 last 233.97)
 - /unlink 26960 (time 1734259114.65 total 109.22 last 237.90)
 - /unlink 29161 (time 1734259124.65 total 119.22 last 220.07)
 - /unlink 30000 (time 1734259128.48 total 123.05 last 218.93)
 - /unlink 32126 (time 1734259138.49 total 133.05 last 212.53)
 - /unlink 34381 (time 1734259148.50 total 143.07 last 225.27)
 - /unlink 36332 (time 1734259158.50 total 153.07 last 195.05)
 - /unlink 38376 (time 1734259168.50 total 163.07 last 204.35)
 - /unlink 40000 (time 1734259175.94 total 170.51 last 218.33)
 - /unlink 42099 (time 1734259185.94 total 180.51 last 209.84)
 - /unlink 44252 (time 1734259195.95 total 190.51 last 215.23)
 - /unlink 46450 (time 1734259205.95 total 200.52 last 219.79)
 - /unlink 48671 (time 1734259215.95 total 210.52 last 222.09)
 - /unlink 50000 (time 1734259221.93 total 216.49 last 222.34)
 - /unlink 52211 (time 1734259231.93 total 226.49 last 221.09)
 - /unlink 54398 (time 1734259241.93 total 236.50 last 218.63)
 - /unlink 56759 (time 1734259251.93 total 246.50 last 236.00)
 - /unlink 59326 (time 1734259261.94 total 256.50 last 256.65)
 - /unlink 60000 (time 1734259264.48 total 259.04 last 265.25)
 - /unlink 62587 (time 1734259274.48 total 269.05 last 258.65)
total: 64768 /unlink in 276.41 seconds: 234.32 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 1517 (time 1734259309.70 total 10.01 last 151.60)
 - open/close 2916 (time 1734259319.70 total 20.01 last 139.88)
 - open/close 4300 (time 1734259329.70 total 30.01 last 138.39)
 - open/close 5488 (time 1734259339.71 total 40.02 last 118.72)
 - open/close 6797 (time 1734259349.71 total 50.02 last 130.89)
 - open/close 8199 (time 1734259359.71 total 60.02 last 140.12)
 - open/close 9469 (time 1734259369.72 total 70.02 last 126.99)
 - open/close 10000 (time 1734259373.67 total 73.98 last 134.13)
 - open/close 11557 (time 1734259383.68 total 83.98 last 155.67)
 - open/close 13180 (time 1734259393.68 total 93.99 last 162.27)
 - open/close 14734 (time 1734259403.68 total 103.99 last 155.32)
 - open/close 16294 (time 1734259413.68 total 113.99 last 155.98)
 - open/close 17999 (time 1734259423.68 total 123.99 last 170.49)
 - open/close 19669 (time 1734259433.69 total 134.00 last 166.92)
 - open/close 20000 (time 1734259435.83 total 136.13 last 154.87)
 - open/close 21561 (time 1734259445.83 total 146.14 last 156.06)
 - open/close 23083 (time 1734259455.83 total 156.14 last 152.19)
 - open/close 24852 (time 1734259465.84 total 166.14 last 176.80)
 - open/close 26687 (time 1734259475.84 total 176.15 last 183.43)
 - open/close 28523 (time 1734259485.84 total 186.15 last 183.57)
 - open/close 30000 (time 1734259493.34 total 193.65 last 196.86)
 - open/close 31878 (time 1734259503.35 total 203.66 last 187.73)
 - open/close 33818 (time 1734259513.35 total 213.66 last 193.96)
 - open/close 35646 (time 1734259523.35 total 223.66 last 182.75)
 - open/close 37524 (time 1734259533.36 total 233.67 last 187.65)
 - open/close 39129 (time 1734259543.37 total 243.67 last 160.40)
 - open/close 40000 (time 1734259549.41 total 249.72 last 144.09)
 - open/close 41367 (time 1734259559.41 total 259.72 last 136.67)
 - open/close 42818 (time 1734259569.42 total 269.72 last 145.05)
 - open/close 44370 (time 1734259579.42 total 279.73 last 155.13)
 - open/close 45795 (time 1734259589.42 total 289.73 last 142.49)
 - open/close 47277 (time 1734259599.43 total 299.73 last 148.14)
 - open/close 48877 (time 1734259609.43 total 309.74 last 159.93)
 - open/close 50000 (time 1734259615.93 total 316.24 last 172.73)
 - open/close 51594 (time 1734259625.93 total 326.24 last 159.36)
 - open/close 53097 (time 1734259635.94 total 336.24 last 150.28)
 - open/close 54736 (time 1734259645.94 total 346.25 last 163.84)
 - open/close 56331 (time 1734259655.94 total 356.25 last 159.46)
 - open/close 57828 (time 1734259665.94 total 366.25 last 149.67)
 - open/close 59443 (time 1734259675.94 total 376.25 last 161.50)
 - open/close 60000 (time 1734259679.68 total 379.98 last 149.25)
 - open/close 61696 (time 1734259689.68 total 389.99 last 169.54)
 - open/close 63320 (time 1734259699.68 total 399.99 last 162.34)
total: 64768 open/close in 409.30 seconds: 158.24 ops/second
 - /unlink 2491 (time 1734259720.49 total 10.00 last 249.03)
 - /unlink 5091 (time 1734259730.49 total 20.00 last 259.99)
 - /unlink 7835 (time 1734259740.49 total 30.00 last 274.35)
 - /unlink 10000 (time 1734259747.42 total 36.93 last 312.67)
 - /unlink 12845 (time 1734259757.42 total 46.93 last 284.42)
 - /unlink 15614 (time 1734259767.42 total 56.94 last 276.81)
 - /unlink 18770 (time 1734259777.42 total 66.94 last 315.56)
 - /unlink 20000 (time 1734259780.85 total 70.36 last 359.08)
 - /unlink 23394 (time 1734259790.85 total 80.36 last 339.39)
 - /unlink 26817 (time 1734259800.85 total 90.36 last 342.25)
 - /unlink 30000 (time 1734259810.83 total 100.34 last 319.09)
 - /unlink 33430 (time 1734259820.83 total 110.34 last 343.00)
 - /unlink 36536 (time 1734259830.83 total 120.34 last 310.56)
 - /unlink 39384 (time 1734259840.83 total 130.34 last 284.70)
 - /unlink 40000 (time 1734259843.09 total 132.60 last 272.62)
 - /unlink 42946 (time 1734259853.09 total 142.61 last 294.50)
 - /unlink 46024 (time 1734259863.09 total 152.61 last 307.77)
 - /unlink 48730 (time 1734259873.10 total 162.61 last 270.54)
 - /unlink 50000 (time 1734259877.79 total 167.30 last 270.62)
 - /unlink 52938 (time 1734259887.79 total 177.30 last 293.74)
 - /unlink 56016 (time 1734259897.79 total 187.31 last 307.74)
 - /unlink 59237 (time 1734259907.79 total 197.31 last 322.08)
 - /unlink 60000 (time 1734259909.84 total 199.35 last 373.83)
 - /unlink 63611 (time 1734259919.84 total 209.35 last 361.04)
total: 64768 /unlink in 212.36 seconds: 304.99 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 2006 (time 1734259950.02 total 10.00 last 200.51)
 - open/close 4109 (time 1734259960.02 total 20.01 last 210.24)
 - open/close 6149 (time 1734259970.03 total 30.01 last 203.88)
 - open/close 8190 (time 1734259980.03 total 40.02 last 204.03)
 - open/close 10000 (time 1734259989.31 total 49.30 last 194.96)
 - open/close 12055 (time 1734259999.32 total 59.30 last 205.49)
 - open/close 14153 (time 1734260009.32 total 69.30 last 209.75)
 - open/close 16186 (time 1734260019.32 total 79.31 last 203.20)
 - open/close 18215 (time 1734260029.32 total 89.31 last 202.90)
 - open/close 20000 (time 1734260038.18 total 98.17 last 201.47)
 - open/close 21959 (time 1734260048.18 total 108.17 last 195.90)
 - open/close 23849 (time 1734260058.19 total 118.17 last 188.92)
 - open/close 25595 (time 1734260068.19 total 128.18 last 174.56)
 - open/close 27231 (time 1734260078.20 total 138.18 last 163.50)
 - open/close 28975 (time 1734260088.20 total 148.18 last 174.37)
 - open/close 30000 (time 1734260093.50 total 153.49 last 193.23)
 - open/close 31752 (time 1734260103.51 total 163.49 last 175.14)
 - open/close 33610 (time 1734260113.51 total 173.49 last 185.78)
 - open/close 35444 (time 1734260123.51 total 183.49 last 183.36)
 - open/close 37448 (time 1734260133.51 total 193.50 last 200.31)
 - open/close 39566 (time 1734260143.51 total 203.50 last 211.79)
 - open/close 40000 (time 1734260145.67 total 205.66 last 200.87)
 - open/close 42085 (time 1734260155.67 total 215.66 last 208.48)
 - open/close 44444 (time 1734260165.68 total 225.66 last 235.87)
 - open/close 46650 (time 1734260175.68 total 235.67 last 220.51)
 - open/close 49151 (time 1734260185.68 total 245.67 last 250.02)
 - open/close 50000 (time 1734260188.94 total 248.93 last 260.29)
 - open/close 52586 (time 1734260198.95 total 258.93 last 258.55)
 - open/close 55248 (time 1734260208.95 total 268.93 last 266.16)
 - open/close 58013 (time 1734260218.95 total 278.93 last 276.49)
 - open/close 60000 (time 1734260225.99 total 285.97 last 282.30)
 - open/close 63071 (time 1734260235.99 total 295.98 last 307.02)
total: 64768 open/close in 302.46 seconds: 214.14 ops/second
 - /unlink 4697 (time 1734260253.52 total 10.00 last 469.62)
 - /unlink 9469 (time 1734260263.52 total 20.00 last 477.17)
 - /unlink 14410 (time 1734260273.52 total 30.00 last 494.04)
 - /unlink 19789 (time 1734260283.52 total 40.00 last 537.90)
 - /unlink 24916 (time 1734260293.52 total 50.00 last 512.64)
 - /unlink 29817 (time 1734260303.52 total 60.00 last 490.08)
 - /unlink 34005 (time 1734260313.52 total 70.01 last 418.73)
 - /unlink 38013 (time 1734260323.53 total 80.01 last 400.71)
 - /unlink 40000 (time 1734260328.36 total 84.85 last 410.60)
 - /unlink 44185 (time 1734260338.37 total 94.85 last 418.44)
 - /unlink 48238 (time 1734260348.37 total 104.85 last 405.28)
 - /unlink 50000 (time 1734260352.02 total 108.50 last 482.59)
 - /unlink 55316 (time 1734260362.02 total 118.50 last 531.58)
 - /unlink 60000 (time 1734260370.09 total 126.57 last 580.52)
total: 64768 /unlink in 135.38 seconds: 478.40 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
 - open/close 2997 (time 1734260411.20 total 10.00 last 299.69)
 - open/close 5610 (time 1734260421.20 total 20.00 last 261.25)
 - open/close 8321 (time 1734260431.21 total 30.01 last 271.01)
 - open/close 10000 (time 1734260438.02 total 36.82 last 246.46)
 - open/close 12279 (time 1734260448.02 total 46.82 last 227.84)
 - open/close 14568 (time 1734260458.02 total 56.82 last 228.84)
 - open/close 16945 (time 1734260468.03 total 66.83 last 237.63)
 - open/close 19004 (time 1734260478.03 total 76.83 last 205.83)
 - open/close 20000 (time 1734260482.27 total 81.07 last 234.65)
 - open/close 22204 (time 1734260492.27 total 91.07 last 220.39)
 - open/close 24407 (time 1734260502.28 total 101.08 last 220.26)
 - open/close 26911 (time 1734260512.28 total 111.08 last 250.20)
 - open/close 29646 (time 1734260522.28 total 121.08 last 273.49)
 - open/close 32701 (time 1734260532.29 total 131.09 last 305.36)
 - open/close 35482 (time 1734260542.29 total 141.09 last 278.09)
 - open/close 38178 (time 1734260552.29 total 151.09 last 269.53)
 - open/close 40000 (time 1734260559.83 total 158.63 last 241.72)
 - open/close 43040 (time 1734260569.83 total 168.63 last 303.91)
 - open/close 46131 (time 1734260579.83 total 178.63 last 309.06)
 - open/close 49283 (time 1734260589.83 total 188.63 last 315.20)
 - open/close 50000 (time 1734260592.86 total 191.66 last 236.85)
 - open/close 52679 (time 1734260602.86 total 201.66 last 267.88)
 - open/close 56025 (time 1734260612.86 total 211.66 last 334.59)
 - open/close 59243 (time 1734260622.86 total 221.66 last 321.73)
 - open/close 60000 (time 1734260625.66 total 224.46 last 270.66)
 - open/close 63887 (time 1734260635.66 total 234.46 last 388.67)
total: 64768 open/close in 236.42 seconds: 273.95 ops/second
 - /unlink 5675 (time 1734260648.52 total 10.00 last 567.46)
 - /unlink 10000 (time 1734260657.13 total 18.61 last 502.52)
 - /unlink 14778 (time 1734260667.13 total 28.61 last 477.80)
 - /unlink 19948 (time 1734260677.13 total 38.61 last 516.96)
 - /unlink 25505 (time 1734260687.13 total 48.61 last 555.66)
 - /unlink 30000 (time 1734260694.87 total 56.35 last 580.78)
 - /unlink 35335 (time 1734260704.87 total 66.35 last 533.42)
 - /unlink 40000 (time 1734260712.79 total 74.27 last 588.99)
 - /unlink 45563 (time 1734260722.79 total 84.27 last 556.25)
 - /unlink 50000 (time 1734260729.90 total 91.37 last 624.73)
 - /unlink 55848 (time 1734260739.90 total 101.37 last 584.73)
 - /unlink 60000 (time 1734260746.36 total 107.84 last 641.96)
total: 64768 /unlink in 115.23 seconds: 562.10 ops/second
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
fail_loc=0
fail_val=0
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
PASS 106 (3035s)

== conf-sanity-1 test 109a: test lctl clear_conf fsname == 06:06:30 (1734260790)

 SKIP: conf-sanity-1 test_109a LU-8727: no implementation for ZFS
SKIP 109a (1s)

== conf-sanity-1 test 109b: test lctl clear_conf one config ========================================================== 06:06:31 (1734260791)

 SKIP: conf-sanity-1 test_109b LU-8727: no implementation for ZFS
SKIP 109b (1s)

== conf-sanity-1 test 116: big size MDT support ========== 06:06:32 (1734260792)

 SKIP: conf-sanity-1 test_116 ldiskfs only test
SKIP 116 (0s)

== conf-sanity-1 test 122a: Check OST sequence update ==== 06:06:32 (1734260792)

 SKIP: conf-sanity-1 test_122a needs >= 2 MDTs
SKIP 122a (1s)

== conf-sanity-1 test 123ah: del_ost cancels config log entries correctly ========================================================== 06:06:33 (1734260793)
Checking servers environments
Checking clients oleg122-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
Starting client oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
Started clients oleg122-client.virtnet: 
192.168.201.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800aae98000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800aae98000.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 4s: want 'procname_uid' got 'procname_uid'
disable quota as required
del_ost: dry run for target lustre-OST0000
config_log: lustre-MDT0000
[DRY RUN] cancel catalog 'lustre-MDT0000:20':"- { index: 20, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-MDT0000:19':"- { index: 19, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.201.122@tcp }"
[DRY RUN] cancel catalog 'lustre-MDT0000:18':"- { index: 18, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-client
[DRY RUN] cancel catalog 'lustre-client:25':"- { index: 25, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-client:24':"- { index: 24, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.201.122@tcp }"
[DRY RUN] cancel catalog 'lustre-client:23':"- { index: 23, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-MDT0000
cancel catalog lustre-MDT0000 log_idx 20: done
cancel catalog lustre-MDT0000 log_idx 19: done
cancel catalog lustre-MDT0000 log_idx 18: done
del_ost: cancelled 3 catalog entries
config_log: lustre-client
cancel catalog lustre-client log_idx 25: done
cancel catalog lustre-client log_idx 24: done
cancel catalog lustre-client log_idx 23: done
del_ost: cancelled 3 catalog entries
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
oleg122-server: rmmod: ERROR: Module lustre is in use
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
fail_loc=0
PASS 123ah (90s)

== conf-sanity-1 test 132: hsm_actions processed after failover ========================================================== 06:08:03 (1734260883)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: 
Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 mdt.hsm_control=enabled
Writing lustre-mdt1/mdt1 properties
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:sys.timeout=20
  lustre:mdt.hsm_control=enabled
  lustre:version=1
  lustre:flags=69
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
PASS 132 (23s)

== conf-sanity-1 test 133: stripe QOS: free space balance in a pool ========================================================== 06:08:26 (1734260906)

 SKIP: conf-sanity-1 test_133 needs >= 4 OSTs
SKIP 133 (1s)

== conf-sanity-1 test 135: check the behavior when changelog is wrapped around ========================================================== 06:08:27 (1734260907)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
oleg122-client: fail_loc=0x1312
oleg122-client: fail_val=5
oleg122-server: fail_loc=0x1312
oleg122-server: fail_val=5
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
Registered 1 changelog users: 'cl1'

Wrap arround changelog catalog
 - open/close 4273 (time 1734260942.40 total 10.00 last 427.27)
total: 4500 open/close in 10.52 seconds: 427.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #12998
total: 4500 /unlink in 5.49 seconds: 819.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
total: 4500 open/close in 9.81 seconds: 458.81 ops/second
total: 4500 /unlink in 4.77 seconds: 943.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
total: 4500 open/close in 9.35 seconds: 481.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
total: 4500 /unlink in 4.36 seconds: 1032.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
total: 4500 open/close in 9.96 seconds: 451.79 ops/second
total: 4500 /unlink in 5.04 seconds: 893.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
total: 4500 open/close in 9.91 seconds: 454.07 ops/second
total: 4500 /unlink in 5.12 seconds: 878.11 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 10973: 22811 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
PASS 135 (133s)

== conf-sanity-1 test 136: don't panic with bad obdecho setup ========================================================== 06:10:40 (1734261040)

 SKIP: conf-sanity-1 test_136 needs >= 2 MDTs
SKIP 136 (1s)

== conf-sanity-1 test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:10:41 (1734261041)

 SKIP: conf-sanity-1 test_137 needs >= 3 MDTs
SKIP 137 (0s)

== conf-sanity-1 test 140: remove_updatelog script actions ========================================================== 06:10:41 (1734261041)

 SKIP: conf-sanity-1 test_140 needs >= 2 MDTs
SKIP 140 (1s)

== conf-sanity-1 test 151a: damaged local config doesn't prevent mounting ========================================================== 06:10:42 (1734261042)

 SKIP: conf-sanity-1 test_151a ldiskfs only test
SKIP 151a (1s)

== conf-sanity-1 test 151b: -ENOSPC doesn't affect mount ========================================================== 06:10:43 (1734261043)

 SKIP: conf-sanity-1 test_151b ldiskfs only test
SKIP 151b (0s)

== conf-sanity-1 test 153a: bypass invalid NIDs quickly == 06:10:44 (1734261044)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
waiting for mount ...
waiting for mount ...
          "192.168.201.122@tcp": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 5 }
          "192.168.252.112@tcp": { connects: 1, replied: 0, uptodate: discovering, sec_ago: 10 }
          "10.252.252.113@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
          "192.168.201.122@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
setup single mount lustre success
umount lustre on /mnt/lustre.....
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
PASS 153a (179s)

== conf-sanity-1 test 153b: added IPv6 NID support ======= 06:13:43 (1734261223)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
Checking servers environments
Checking clients oleg122-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
Starting client oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
Started clients oleg122-client.virtnet: 
192.168.201.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b583f800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b583f800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
disable quota as required
PASS 153b (69s)

== conf-sanity-1 test 153c: don't stuck on unreached NID ========================================================== 06:14:52 (1734261292)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
start mds service on oleg122-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg122-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
          "192.168.252.112@tcp": { connects: 3, replied: 0, uptodate: discovering, sec_ago: 0 }
          "192.168.252.113@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 10 }
          "192.168.252.115@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 5 }
connection attempts: 7
Waiting for mount to fail
mount.lustre: mount 192.168.252.112@tcp:192.168.252.113@tcp:192.168.252.115@tcp:/lustre at /mnt/lustre failed: Input/output error
Is the MGS running?
umount lustre on /mnt/lustre.....
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
PASS 153c (319s)

== conf-sanity-1 test 154: expand .. on rename after MDT backup restore ========================================================== 06:20:11 (1734261611)

 SKIP: conf-sanity-1 test_154 ldiskfs only test
SKIP 154 (1s)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
== conf-sanity-1 test complete, duration 6529 sec ======== 06:20:31 (1734261631)
=== conf-sanity-1: start cleanup 06:20:32 (1734261632) ===
=== conf-sanity-1: finish cleanup 06:20:33 (1734261633) ===