-----============= acceptance-small: conf-sanity-1 ============----- Sun Dec 15 04:31:35 EST 2024
mgs: CentOS Linux release 7.9.2009 (Core)
MGS_OS_VERSION_ID=7
MGS_OS_ID=centos
MGS_OS_VERSION_CODE=117440512
MGS_OS_ID_LIKE=rhel fedora centos
mds1: CentOS Linux release 7.9.2009 (Core)
MDS1_OS_ID_LIKE=rhel fedora centos
MDS1_OS_ID=centos
MDS1_OS_VERSION_ID=7
MDS1_OS_VERSION_CODE=117440512
ost1: CentOS Linux release 7.9.2009 (Core)
OST1_OS_VERSION_CODE=117440512
OST1_OS_VERSION_ID=7
OST1_OS_ID_LIKE=rhel fedora centos
OST1_OS_ID=centos
client: CentOS Linux release 7.9.2009 (Core)
CLIENT_OS_ID=centos
CLIENT_OS_ID_LIKE=rhel fedora centos
CLIENT_OS_VERSION_ID=7
CLIENT_OS_VERSION_CODE=117440512
oleg312-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity-1.*ex: No such file or directory
excepting tests: 
skipping tests SLOW=no: 106
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg312-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server

== conf-sanity-1 test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 04:33:18 (1734255198)
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
Remove mds config log
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
start ost1 service on oleg312-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: No such file or directory
oleg312-server: Is the MGS specification correct?
oleg312-server: Is the filesystem name correct?
oleg312-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
Start of /dev/mapper/mds1_flakey on mds1 failed 2
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 17 (161s)

== conf-sanity-1 test 18: check mkfs creates large journals ========================================================== 04:36:00 (1734255360)
use device /dev/mapper/mds1_flakey with MIN=2000000
mount mds with large journal...
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
mount lustre system...
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
setup single mount lustre success
check journal size...
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Success: mkfs creates large journals. Size: 78M
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 18 (160s)

== conf-sanity-1 test 21c: start mds between two osts, stop mds last ========================================================== 04:38:40 (1734255520)
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost2 service on oleg312-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop ost2 service on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

PASS 21c (108s)

== conf-sanity-1 test 29: permanently remove an OST ====== 04:40:28 (1734255628)
start ost2 service on oleg312-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0001
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Setting lustre-OST0001.osc.active from 1 to 0
Waiting 90s for '0'
Updated after 9s: want '0' got '0'
Live client success: got lustre-OST0001_UUID	IDLE	DEACTIVATED
osc.lustre-OST0001-osc-MDT0000.active updated after 0 sec (got 0)
osc.lustre-OST0001-osc-MDT0001.active updated after 0 sec (got 0)
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
New client success: got 'lustre-OST0001_UUID	NEW	DEACTIVATED'
Setting lustre-OST0001.osc.active from 0 to 1
Waiting 90s for '1'
Updated after 5s: want '1' got '1'
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 29 (166s)

== conf-sanity-1 test 35a: Reconnect to the last active server first ========================================================== 04:43:14 (1734255794)
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
debug=ha
Set up a fake failnode for the MDS
Wait for RECONNECT_INTERVAL seconds (10s)
conf-sanity.sh test_35a 2024-12-15 4h43m45s
Stopping the MDT: lustre-MDT0000
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Restarting the MDT: lustre-MDT0000
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
Wait for df (8983) ... 
done
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout
Debug log: 102 lines, 102 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 35a (134s)

== conf-sanity-1 test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:45:28 (1734255928)
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
debug=ha
conf-sanity.sh test_35b 2024-12-15 4h46m03s
Set up a fake failnode for the MDS
at_max=0
at_max=0
Injecting EBUSY on MDS
fail_loc=0x80000136
mdc.lustre-MDT0000-mdc-ffff8800b6156800.stats=clear
mdc.lustre-MDT0001-mdc-ffff8800b6156800.stats=clear
Creating a test file and stat it
  File: '/mnt/lustre/d35b.conf-sanity-1/f35b.conf-sanity-1'
  Size: 0         	Blocks: 0          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d	Inode: 144115238826934274  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:46:49.000000000 -0500
Modify: 2024-12-15 04:46:49.000000000 -0500
Change: 2024-12-15 04:46:49.000000000 -0500
 Birth: -
Stop injecting EBUSY on MDS
fail_loc=0
done
at_max=600
at_max=600
Debug log: 76 lines, 76 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 35b (165s)

== conf-sanity-1 test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:48:13 (1734256093)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov -o nosvc -n  /dev/mapper/mds1_flakey /mnt/lustre-mds1
nomtab: 1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
Start /dev/mapper/mds1_flakey without service
Commit the device label on /dev/mapper/mds1_flakey
Started lustre:MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg312-server
Starting mds1: -o localrecov -o nomgs,force  /dev/mapper/mds1_flakey /mnt/lustre-mds1
force: 1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
blah blah
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
PASS 41b (86s)

== conf-sanity-1 test 46a: handle ost additional - wide striped file ========================================================== 04:49:39 (1734256179)
Testing with 2 OSTs
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg312-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b40b2000.ost_server_uuid 50
osc.lustre-OST0001-osc-ffff8800b40b2000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre2.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre2
/mnt/lustre2
stripe_count:  -1 stripe_size:   4194304 pattern:       raid0 stripe_offset: -1

/mnt/lustre2/f41b.conf-sanity-1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	             2	          0x2	   0x240000401

/mnt/lustre2/widestripe
lmm_stripe_count:  2
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 1
	obdidx		 objid		 objid		 group
	     1	             2	          0x2	   0x2c0000401
	     0	            34	         0x22	   0x240000401

  File: '/mnt/lustre/widestripe'
  Size: 3         	Blocks: 1          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115238826934273  Links: 1
Access: (0674/-rw-rwxr--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:50:10.000000000 -0500
Modify: 2024-12-15 04:50:10.000000000 -0500
Change: 2024-12-15 04:50:11.000000000 -0500
 Birth: -
umount lustre on /mnt/lustre2.....
Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:)
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
stop ost1 service on oleg312-server
stop mds service on oleg312-server
stop mds service on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 46a (138s)

== conf-sanity-1 test 47a: server restart does not lose lru_resize ========================================================== 04:51:57 (1734256317)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
setup single mount lustre success
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9281800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a9281800.lru_size=100
ldlm.namespaces.lustre-OST0000-osc-ffff8800a9281800.lru_size=100
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9281800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a9281800.lru_size=100
Failing ost1 on oleg312-server
Stopping /mnt/lustre-ost1 (opts:) on oleg312-server
04:52:49 (1734256369) shut down
facet: ost1 facet_host: oleg312-server facet_failover_host: oleg312-server
Failover ost1 to oleg312-server
mount facets: ost1
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
04:53:04 (1734256384) targets are mounted
04:53:04 (1734256384) facet_failover done
Failing mds1 on oleg312-server
Stopping /mnt/lustre-mds1 (opts:) on oleg312-server
04:53:05 (1734256385) shut down
facet: mds1 facet_host: oleg312-server facet_failover_host: oleg312-server
Failover mds1 to oleg312-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
04:53:20 (1734256400) targets are mounted
04:53:20 (1734256400) facet_failover done
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9281800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a9281800.lru_size=100
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 47a (110s)

== conf-sanity-1 test 47b: client restart does not lose lru_resize ========================================================== 04:53:48 (1734256428)
Failing mds1 on oleg312-server
04:53:49 (1734256429) shut down
facet: mds1 facet_host: oleg312-server facet_failover_host: oleg312-server
Failover mds1 to oleg312-server
mount facets: mds1
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
04:54:12 (1734256452) targets are mounted
04:54:12 (1734256452) facet_failover done
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 95
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 95
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
setup single mount lustre success
ln: failed to create symbolic link '/home/green/git/lustre-release/lustre/utils/lctl': File exists

 SKIP: conf-sanity-1 test_47b Can not create symlink /usr/sbin/lctl
SKIP 47b (61s)

== conf-sanity-1 test 48: too many acls on file ========== 04:54:48 (1734256488)
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg312-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
mount.lustre: according to /etc/mtab 192.168.203.112@tcp:/lustre is already mounted on /mnt/lustre
setup single mount lustre success
debug=0
debug=0
/mnt/lustre
stripe_count:  -1 stripe_size:   4194304 pattern:       raid0 stripe_offset: -1

/mnt/lustre/widestripe
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	             4	          0x4	   0x280000401

It is expected to hold at least 4500 ACL entries
  File: '/mnt/lustre/widestripe'
  Size: 3         	Blocks: 8          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115205289279491  Links: 1
Access: (0664/-rw-rw-r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:54:53.000000000 -0500
Modify: 2024-12-15 04:54:53.000000000 -0500
Change: 2024-12-15 04:55:57.000000000 -0500
 Birth: -
getfacl: Removing leading '/' from absolute path names
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg312-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 48 (150s)

== conf-sanity-1 test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:57:18 (1734256638)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
setup single mount lustre success
check ldlm_timout...
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 49a (57s)

== conf-sanity-1 test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:58:15 (1734256695)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 49b (75s)

== conf-sanity-1 test 50c: lazystatfs one server down ==== 04:59:30 (1734256770)
start mds service on oleg312-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg312-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800b36fe000.lazystatfs=1
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
oleg312-server: oleg312-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7612
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        95248        1680       84912   2% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        95248        1532       85060   2% /mnt/lustre[MDT:1]
lustre-OST0001_UUID       142216        1528      126688   2% /mnt/lustre[OST:1]

filesystem_summary:       142216        1528      126688   2% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (30996) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19

PASS 50c (94s)

== conf-sanity-1 test 50d: lazystatfs client/server conn race ========================================================== 05:01:04 (1734256864)
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg312-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0001
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800b36f8800.lazystatfs=1
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7612
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        95248        1680       84912   2% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        95248        1532       85060   2% /mnt/lustre[MDT:1]
lustre-OST0001_UUID       142216        1528      126688   2% /mnt/lustre[OST:1]

filesystem_summary:       142216        1528      126688   2% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (1405) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19

PASS 50d (75s)

== conf-sanity-1 test 50f: normal statfs one server in down ========================================================== 05:02:19 (1734256939)
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
start ost2 service on oleg312-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0001
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
oleg312-server: oleg312-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800b58de000.lazystatfs=0
multiop /mnt/lustre v_f
TMPPIPE=/tmp/multiop_open_wait_pipe.7612
start ost2 service on oleg312-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0001
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19

PASS 50f (120s)

== conf-sanity-1 test 50g: deactivated OST should not cause panic ========================================================== 05:04:19 (1734257059)
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
start ost2 service on oleg312-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0001
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg312-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b379d000.ost_server_uuid 50
osc.lustre-OST0001-osc-ffff8800b379d000.ost_server_uuid in FULL state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Filesystem                  1K-blocks  Used Available Use% Mounted on
192.168.203.112@tcp:/lustre    142216  1524    126692   2% /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 sys.ldlm_timeout=19

PASS 50g (72s)

== conf-sanity-1 test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:05:31 (1734257131)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
setup single mount lustre success
fail_loc=0x142
start ost2 service on oleg312-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 51 (187s)

== conf-sanity-1 test 54a: test llverdev and partial verify of device ========================================================== 05:08:38 (1734257318)
oleg312-server: oleg312-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p
oleg312-server: llverdev: /dev/mapper/ost1_flakey is 4294967296 bytes (4 GB) in size
oleg312-server: timestamp: 1734257321 chunksize: 1048576 size: 4294967296
oleg312-server: write offset:        4194304kB   inf MB/s            
oleg312-server: read offset:        4194304kB   inf MB/s            
oleg312-server: llverdev: data verified successfully
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 54a (71s)

== conf-sanity-1 test 55: check lov_objid size =========== 05:09:49 (1734257389)

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre:OST03ff
Index:      1023
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST03ff
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST03ff -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: mount.lustre: mount /dev/mapper/mds2_flakey at /mnt/lustre-mds2 failed: No such file or directory
oleg312-server: Is the MGS specification correct?
oleg312-server: Is the filesystem name correct?
oleg312-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
Start of /dev/mapper/mds2_flakey on mds2 failed 2
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST03ff-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST03ff
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03ff-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping client oleg312-client.virtnet /mnt/lustre opts:
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: mount.lustre: mount /dev/mapper/mds2_flakey at /mnt/lustre-mds2 failed: No such file or directory
oleg312-server: Is the MGS specification correct?
oleg312-server: Is the filesystem name correct?
oleg312-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
Start of /dev/mapper/mds2_flakey on mds2 failed 2
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST03ff-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST03ff
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03ff-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
checking size of lov_objid for ost index 1023
/home/green/git/lustre-release/lustre/tests/conf-sanity-1.sh: line 841: lov_objid_size: command not found
/home/green/git/lustre-release/lustre/tests/conf-sanity-1.sh: line 841: [: 8192: unary operator expected
ok, lov_objid size is correct: 8192
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping client oleg312-client.virtnet /mnt/lustre opts:
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre:OST0800
Index:      2048
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST0800
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0800 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: mount.lustre: mount /dev/mapper/mds2_flakey at /mnt/lustre-mds2 failed: No such file or directory
oleg312-server: Is the MGS specification correct?
oleg312-server: Is the filesystem name correct?
oleg312-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
Start of /dev/mapper/mds2_flakey on mds2 failed 2
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0800-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0800
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0800-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping client oleg312-client.virtnet /mnt/lustre opts:
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: mount.lustre: mount /dev/mapper/mds2_flakey at /mnt/lustre-mds2 failed: No such file or directory
oleg312-server: Is the MGS specification correct?
oleg312-server: Is the filesystem name correct?
oleg312-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
Start of /dev/mapper/mds2_flakey on mds2 failed 2
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0800-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0800
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0800-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
checking size of lov_objid for ost index 2048
/home/green/git/lustre-release/lustre/tests/conf-sanity-1.sh: line 841: lov_objid_size: command not found
/home/green/git/lustre-release/lustre/tests/conf-sanity-1.sh: line 841: [: 16392: unary operator expected
ok, lov_objid size is correct: 16392
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping client oleg312-client.virtnet /mnt/lustre opts:
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 55 (215s)

== conf-sanity-1 test 56a: check big OST indexes and out-of-index-order start ========================================================== 05:13:24 (1734257604)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey

   Permanent disk data:
Target:     lustre:OST2710
Index:      10000
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST2710
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST2710 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre:OST03e8
Index:      1000
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost2_flakey
	target name   lustre:OST03e8
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST03e8 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost2_flakey 200000k
Writing CONFIGS/mountdata
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST2710-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST2710
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg312-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST03e8-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST03e8
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
ok
OBDS:
1000: lustre-OST03e8_UUID ACTIVE
10000: lustre-OST2710_UUID ACTIVE
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.00545785 s, 750 kB/s
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping client oleg312-client.virtnet /mnt/lustre opts:
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 56a (155s)

== conf-sanity-1 test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:15:59 (1734257759)

 SKIP: conf-sanity-1 test_56b needs >= 3 MDTs
SKIP 56b (1s)

== conf-sanity-1 test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:16:00 (1734257760)
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x142
              (OST update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 failover.node=192.168.203.112@tcp

Writing CONFIGS/mountdata
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg312-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Cannot assign requested address
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 99
oleg312-server: error: set_param: param_path 'seq/cli-lustre': No such file or directory
oleg312-server: error: set_param: setting 'seq/cli-lustre'='OST0000-super.width=65536': No such file or directory
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 99
umount lustre on /mnt/lustre.....
stop ost1 service on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 57a (41s)

== conf-sanity-1 test 57b: initial registration from servicenode should not fail ========================================================== 05:16:41 (1734257801)
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 failover.node=192.168.203.112@tcp


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 failover.node=192.168.203.112@tcp

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x142
              (OST update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x142
              (OST update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 failover.node=192.168.203.112@tcp


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1142
              (OST update writeconf no_primnode )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20  failover.node=192.168.203.112@tcp:192.168.203.112@tcp

Writing CONFIGS/mountdata
start mds service on oleg312-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 57b (62s)

== conf-sanity-1 test 59: writeconf mount option ========= 05:17:43 (1734257863)
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
original ost count: 1 (expect > 0)
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
after mdt writeconf count: 0 (expect 0)
OST start without writeconf should fail:
oleg312-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: No such file or directory
oleg312-server: Is the MGS specification correct?
oleg312-server: Is the filesystem name correct?
oleg312-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg312-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
OST start with writeconf should succeed:
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
after ost writeconf count: 1 (expect 1)
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
after ost2 writeconf count: 2 (expect 2)
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1002
              (OST no_primnode )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20  failover.node=192.168.203.112@tcp:192.168.203.112@tcp


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1102
              (OST writeconf no_primnode )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20  failover.node=192.168.203.112@tcp:192.168.203.112@tcp

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 59 (176s)

== conf-sanity-1 test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 05:20:39 (1734258039)

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options          -I 1024 -i 2560 -q -O ^uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000    -I 1024 -i 2560 -q -O ^uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre:MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds2_flakey
	target name   lustre:MDT0001
	kilobytes     200000
	options          -I 1024 -i 2560 -q -O ^uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0001    -I 1024 -i 2560 -q -O ^uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/mapper/mds2_flakey 200000k
Writing CONFIGS/mountdata
oleg312-server: dumpe2fs 1.46.2.wc5 (26-Mar-2022)
stop mds service on oleg312-server
stop mds service on oleg312-server
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 60a (84s)

== conf-sanity-1 test 62: start with disabled journal ==== 05:22:04 (1734258124)
disable journal for mds
tune2fs 1.46.2.wc5 (26-Mar-2022)
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Invalid argument
oleg312-server: This may have multiple causes.
oleg312-server: Are the mount options correct?
oleg312-server: Check the syslog for more info.
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 22
Start of /dev/mapper/mds1_flakey on mds1 failed 22
disable journal for ost
tune2fs 1.46.2.wc5 (26-Mar-2022)
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg312-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Invalid argument
oleg312-server: This may have multiple causes.
oleg312-server: Are the mount options correct?
oleg312-server: Check the syslog for more info.
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 22
oleg312-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg312-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 22
umount lustre on /mnt/lustre.....
stop ost1 service on oleg312-server
stop mds service on oleg312-server
stop mds service on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 62 (112s)

== conf-sanity-1 test 64: check lfs df --lazy ============ 05:23:55 (1734258235)
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg312-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
stop ost2 service on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
/home/green/git/lustre-release/lustre/utils/lfs df
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        95248        1668       84924   2% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        95248        1532       85060   2% /mnt/lustre[MDT:1]
lustre-OST0000_UUID       142216        1524      126692   2% /mnt/lustre[OST:0]

filesystem_summary:       142216        1524      126692   2% /mnt/lustre

umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

oleg312-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg312-server: Is the ldiskfs module available?
oleg312-server: 
oleg312-server: tunefs.lustre FATAL: failed to write local files
oleg312-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 64 (165s)

== conf-sanity-1 test 66: replace nids =================== 05:26:40 (1734258400)
client=34615898 MDS=34615898 OSS=34615898
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Setting lustre-OST0000.osc.active from 1 to 0
Waiting 90s for '0'
Updated after 7s: want '0' got '0'
replace_nids should fail if MDS, OSTs and clients are UP
oleg312-server: error: replace_nids: Operation now in progress
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 115
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
replace_nids should fail if MDS and OSTs are UP
oleg312-server: error: replace_nids: Operation now in progress
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 115
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
replace_nids should fail if MDS is UP
oleg312-server: error: replace_nids: Operation now in progress
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 115
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
start mds service on oleg312-server
Starting mds1: -o localrecov -o nosvc  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
Start /dev/mapper/mds1_flakey without service
Started lustre-MDT0000
command should accept two parameters
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
correct device name should be passed
oleg312-server: error: replace_nids: No such device or address
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 6
wrong nids list should not destroy the system
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
replace OST nid
command should accept two parameters
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
wrong nids list should not destroy the system
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
set NIDs with failover
replace MDS nid
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Setting lustre-OST0000.osc.active from 0 to 1
Waiting 90s for '1'
Waiting 70s for '1'
Updated after 27s: want '1' got '1'
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 66 (239s)

== conf-sanity-1 test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 05:30:39 (1734258639)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST0000
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid

   Permanent disk data:
Target:     lustre:MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds2_flakey
	target name   lustre:MDT0001
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0001 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds2_flakey 200000k
Writing CONFIGS/mountdata
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-server: oleg312-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50
oleg312-server: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50
oleg312-server: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
umount lustre on /mnt/lustre.....
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 70e (113s)

== conf-sanity-1 test 75: The order of --index should be irrelevant ========================================================== 05:32:32 (1734258752)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre-MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x42
              (OST update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre-OST0000
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre-MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x42
              (OST update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre-OST0000
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 75 (84s)

== conf-sanity-1 test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:33:56 (1734258836)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format ost1: /dev/mapper/ost1_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=131072
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
create test files
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        83240        1616       73832   3% /mnt/lustre[MDT:0]
lustre-OST0000_UUID       124712        1388      110724   2% /mnt/lustre[OST:0]

filesystem_summary:       124712        1388      110724   2% /mnt/lustre

UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        72000         272       71728   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID        45008         302       44706   1% /mnt/lustre[OST:0]

filesystem_summary:        44978         272       44706   1% /mnt/lustre

1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0530287 s, 19.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0542736 s, 19.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0657231 s, 16.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0510878 s, 20.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0588906 s, 17.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0541923 s, 19.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.056458 s, 18.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.107497 s, 9.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0524398 s, 20.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0482918 s, 21.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0501497 s, 20.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0498709 s, 21.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0555069 s, 18.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0526444 s, 19.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0730019 s, 14.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0633519 s, 16.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0556048 s, 18.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0553367 s, 18.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.055584 s, 18.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.085956 s, 12.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0662521 s, 15.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0572903 s, 18.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0636002 s, 16.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0641732 s, 16.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0634723 s, 16.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.060802 s, 17.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0650053 s, 16.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0492458 s, 21.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0576178 s, 18.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0519705 s, 20.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0613942 s, 17.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0539326 s, 19.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0539921 s, 19.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0561502 s, 18.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0564252 s, 18.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0457718 s, 22.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0721785 s, 14.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0645211 s, 16.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0652182 s, 16.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0717639 s, 14.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0745609 s, 14.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.066379 s, 15.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0611897 s, 17.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0630563 s, 16.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0561013 s, 18.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0500563 s, 20.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.06039 s, 17.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.118074 s, 8.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0662956 s, 15.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0791831 s, 13.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0514857 s, 20.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0491808 s, 21.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0580711 s, 18.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0515579 s, 20.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0693533 s, 15.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0505636 s, 20.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0536937 s, 19.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0609384 s, 17.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0544934 s, 19.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0636358 s, 16.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0568686 s, 18.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0777411 s, 13.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0641898 s, 16.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0500374 s, 21.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0699157 s, 15.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0951494 s, 11.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0689401 s, 15.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0814279 s, 12.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0879342 s, 11.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0595059 s, 17.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0621634 s, 16.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0602298 s, 17.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0553612 s, 18.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0937897 s, 11.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0686771 s, 15.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0759984 s, 13.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0545001 s, 19.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0757772 s, 13.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0708237 s, 14.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.169207 s, 6.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0647424 s, 16.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0566839 s, 18.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0613989 s, 17.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0596105 s, 17.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0587304 s, 17.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0468274 s, 22.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0987322 s, 10.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0739518 s, 14.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0562949 s, 18.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0631326 s, 16.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0542493 s, 19.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0546077 s, 19.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0600789 s, 17.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0612297 s, 17.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0510241 s, 20.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0788525 s, 13.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.691269 s, 1.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.054017 s, 19.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.106828 s, 9.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0597247 s, 17.6 MB/s
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg312-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg312-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 3)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 24032 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48044 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48045 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48046 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48047 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48048 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48054 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] Pass 1: Memory used: 272k/0k (141k/132k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 154.97MB/s
[Thread 0] Scanned group range [0, 3), inodes 373
Pass 2: Checking directory structure
Pass 2: Memory used: 272k/0k (95k/178k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 99.35MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 272k/0k (95k/178k), time:  0.03/ 0.00/ 0.02
Pass 3A: Memory used: 272k/0k (95k/178k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 272k/0k (93k/180k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 6172.84MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 272k/0k (67k/206k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 272k/0k (67k/206k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 235.07MB/s

         372 inodes used (0.52%, out of 72000)
           5 non-contiguous files (1.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
       22546 blocks used (50.10%, out of 45000)
           0 bad blocks
           1 large file

         244 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         362 files
Memory used: 272k/0k (66k/207k), time:  0.04/ 0.01/ 0.02
I/O read: 1MB, write: 1MB, rate: 22.37MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg312-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg312-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 2)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 22562 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 22563 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] Pass 1: Memory used: 264k/0k (132k/133k), time:  0.02/ 0.01/ 0.00
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 275.44MB/s
[Thread 0] Scanned group range [0, 2), inodes 398
Pass 2: Checking directory structure
Pass 2: Memory used: 264k/0k (86k/179k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 89.93MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 264k/0k (92k/173k), time:  0.04/ 0.02/ 0.01
Pass 3A: Memory used: 264k/0k (92k/173k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 264k/0k (84k/181k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 6097.56MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 264k/0k (65k/200k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 264k/0k (64k/201k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 433.09MB/s

         398 inodes used (0.88%, out of 45008)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 392
       37721 blocks used (83.82%, out of 45000)
           0 bad blocks
           1 large file

         216 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         388 files
Memory used: 264k/0k (64k/201k), time:  0.06/ 0.02/ 0.01
I/O read: 2MB, write: 1MB, rate: 34.32MB/s
oleg312-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/mds1_flakey to 640000 (4k) blocks.
The filesystem on /dev/mapper/mds1_flakey is now 640000 (4k) blocks long.

oleg312-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/ost1_flakey to 1048576 (4k) blocks.
The filesystem on /dev/mapper/ost1_flakey is now 1048576 (4k) blocks long.

e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg312-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg312-server: Use max possible thread num: 2 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 16)
[Thread 1] Scan group range [16, 33)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 1] jumping to group 16
[Thread 1] group 17 finished
[Thread 1] group 18 finished
[Thread 1] group 19 finished
[Thread 1] group 20 finished
[Thread 1] group 21 finished
[Thread 1] group 22 finished
[Thread 1] group 23 finished
[Thread 1] group 24 finished
[Thread 1] group 25 finished
[Thread 1] group 26 finished
[Thread 1] group 27 finished
[Thread 1] group 28 finished
[Thread 1] group 29 finished
[Thread 1] group 30 finished
[Thread 1] group 31 finished
[Thread 1] group 32 finished
[Thread 1] group 33 finished
[Thread 0] group 1 finished
[Thread 1] Pass 1: Memory used: 680k/0k (444k/237k), time:  0.00/ 0.00/ 0.00
[Thread 1] Pass 1: I/O read: 1MB, write: 0MB, rate: 2518.89MB/s
[Thread 1] Scanned group range [16, 33), inodes 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 24032 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48044 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48045 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48046 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48047 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48048 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48054 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] Pass 1: Memory used: 688k/0k (354k/334k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 89.81MB/s
[Thread 0] Scanned group range [0, 16), inodes 373
Pass 2: Checking directory structure
Pass 2: Memory used: 632k/0k (199k/434k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 125.52MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 632k/0k (200k/433k), time:  0.07/ 0.04/ 0.00
Pass 3A: Memory used: 632k/0k (200k/433k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 632k/0k (198k/435k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 1283.70MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 632k/0k (71k/562k), time:  0.04/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 632k/0k (70k/563k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 123.95MB/s

         372 inodes used (0.05%, out of 792000)
           5 non-contiguous files (1.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
      202726 blocks used (31.68%, out of 640000)
           0 bad blocks
           1 large file

         244 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         362 files
Memory used: 632k/0k (69k/564k), time:  0.13/ 0.07/ 0.00
I/O read: 1MB, write: 1MB, rate: 7.87MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg312-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg312-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 32)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 22562 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 22563 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] group 19 finished
[Thread 0] group 20 finished
[Thread 0] group 21 finished
[Thread 0] group 22 finished
[Thread 0] group 23 finished
[Thread 0] group 24 finished
[Thread 0] group 25 finished
[Thread 0] group 26 finished
[Thread 0] group 27 finished
[Thread 0] group 28 finished
[Thread 0] group 29 finished
[Thread 0] group 30 finished
[Thread 0] group 31 finished
[Thread 0] group 32 finished
[Thread 0] Pass 1: Memory used: 468k/0k (344k/125k), time:  0.02/ 0.01/ 0.00
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 304.28MB/s
[Thread 0] Scanned group range [0, 32), inodes 398
Pass 2: Checking directory structure
Pass 2: Memory used: 680k/0k (298k/383k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 89.56MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 680k/0k (298k/383k), time:  0.06/ 0.02/ 0.01
Pass 3A: Memory used: 680k/0k (299k/382k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 680k/0k (296k/385k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 538.50MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 564k/0k (66k/499k), time:  0.03/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 564k/0k (65k/500k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 170.30MB/s

         398 inodes used (0.06%, out of 720128)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 392
      128315 blocks used (12.24%, out of 1048576)
           0 bad blocks
           1 large file

         216 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         388 files
Memory used: 564k/0k (64k/501k), time:  0.10/ 0.04/ 0.01
I/O read: 2MB, write: 1MB, rate: 20.37MB/s
start mds service on oleg312-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=131072
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
check files after expanding the MDT and OST filesystems
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-1 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-1 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-2 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-2 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-3 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-3 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-4 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-4 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-5 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-5 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-6 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-6 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-7 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-7 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-8 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-8 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-9 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-9 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-10 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-10 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-11 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-11 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-12 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-12 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-13 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-13 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-14 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-14 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-15 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-15 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-16 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-16 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-17 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-17 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-18 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-18 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-19 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-19 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-20 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-20 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-21 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-21 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-22 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-22 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-23 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-23 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-24 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-24 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-25 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-25 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-26 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-26 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-27 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-27 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-28 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-28 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-29 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-29 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-30 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-30 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-31 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-31 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-32 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-32 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-33 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-33 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-34 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-34 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-35 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-35 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-36 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-36 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-37 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-37 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-38 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-38 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-39 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-39 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-40 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-40 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-41 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-41 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-42 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-42 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-43 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-43 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-44 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-44 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-45 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-45 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-46 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-46 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-47 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-47 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-48 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-48 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-49 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-49 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-50 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-50 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-51 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-51 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-52 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-52 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-53 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-53 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-54 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-54 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-55 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-55 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-56 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-56 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-57 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-57 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-58 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-58 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-59 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-59 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-60 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-60 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-61 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-61 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-62 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-62 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-63 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-63 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-64 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-64 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-65 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-65 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-66 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-66 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-67 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-67 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-68 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-68 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-69 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-69 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-70 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-70 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-71 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-71 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-72 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-72 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-73 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-73 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-74 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-74 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-75 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-75 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-76 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-76 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-77 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-77 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-78 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-78 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-79 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-79 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-80 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-80 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-81 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-81 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-82 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-82 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-83 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-83 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-84 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-84 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-85 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-85 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-86 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-86 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-87 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-87 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-88 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-88 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-89 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-89 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-90 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-90 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-91 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-91 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-92 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-92 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-93 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-93 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-94 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-94 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-95 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-95 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-96 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-96 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-97 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-97 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-98 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-98 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-99 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-99 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-100 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-100 has size 1048576 OK
create more files after expanding the MDT and OST filesystems
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0769978 s, 13.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0715845 s, 14.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0537645 s, 19.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0619083 s, 16.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0743053 s, 14.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0756451 s, 13.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0755473 s, 13.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.111633 s, 9.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0823286 s, 12.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0542991 s, 19.3 MB/s
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg312-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg312-server: Use max possible thread num: 2 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 16)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 24032 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48044 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48045 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48046 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48047 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48048 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48054 badness 0 to 2 for 10084
[Thread 1] Scan group range [16, 33)
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] Pass 1: Memory used: 500k/0k (278k/223k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 139.31MB/s
[Thread 0] Scanned group range [0, 16), inodes 383
[Thread 1] jumping to group 16
[Thread 1] group 17 finished
[Thread 1] group 18 finished
[Thread 1] group 19 finished
[Thread 1] group 20 finished
[Thread 1] group 21 finished
[Thread 1] group 22 finished
[Thread 1] group 23 finished
[Thread 1] group 24 finished
[Thread 1] group 25 finished
[Thread 1] group 26 finished
[Thread 1] group 27 finished
[Thread 1] group 28 finished
[Thread 1] group 29 finished
[Thread 1] group 30 finished
[Thread 1] group 31 finished
[Thread 1] group 32 finished
[Thread 1] group 33 finished
[Thread 1] Pass 1: Memory used: 592k/0k (307k/286k), time:  0.00/ 0.00/ 0.00
[Thread 1] Pass 1: I/O read: 1MB, write: 0MB, rate: 1434.72MB/s
[Thread 1] Scanned group range [16, 33), inodes 0
Pass 2: Checking directory structure
Pass 2: Memory used: 592k/0k (197k/396k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 170.04MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 592k/0k (198k/395k), time:  0.07/ 0.04/ 0.00
Pass 3A: Memory used: 592k/0k (198k/395k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 592k/0k (196k/397k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 526.59MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 592k/0k (69k/524k), time:  0.03/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 592k/0k (68k/525k), time:  0.01/ 0.01/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 174.16MB/s

         382 inodes used (0.05%, out of 792000)
           5 non-contiguous files (1.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
      202726 blocks used (31.68%, out of 640000)
           0 bad blocks
           1 large file

         254 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         372 files
Memory used: 592k/0k (67k/526k), time:  0.11/ 0.07/ 0.00
I/O read: 1MB, write: 1MB, rate: 9.31MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg312-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg312-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 32)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 22562 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 22563 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] group 19 finished
[Thread 0] group 20 finished
[Thread 0] group 21 finished
[Thread 0] group 22 finished
[Thread 0] group 23 finished
[Thread 0] group 24 finished
[Thread 0] group 25 finished
[Thread 0] group 26 finished
[Thread 0] group 27 finished
[Thread 0] group 28 finished
[Thread 0] group 29 finished
[Thread 0] group 30 finished
[Thread 0] group 31 finished
[Thread 0] group 32 finished
[Thread 0] Pass 1: Memory used: 472k/0k (344k/129k), time:  0.02/ 0.01/ 0.00
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 248.35MB/s
[Thread 0] Scanned group range [0, 32), inodes 402
Pass 2: Checking directory structure
Pass 2: Memory used: 684k/0k (298k/387k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 91.96MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 684k/0k (299k/386k), time:  0.07/ 0.04/ 0.00
Pass 3A: Memory used: 684k/0k (299k/386k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 684k/0k (296k/389k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 6711.41MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 568k/0k (67k/502k), time:  0.02/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 568k/0k (65k/504k), time:  0.01/ 0.01/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 133.48MB/s

         402 inodes used (0.06%, out of 720128)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 396
      130875 blocks used (12.48%, out of 1048576)
           0 bad blocks
           1 large file

         220 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         392 files
Memory used: 568k/0k (64k/505k), time:  0.11/ 0.06/ 0.00
I/O read: 2MB, write: 1MB, rate: 18.86MB/s
oleg312-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/mds1_flakey to 377837 (4k) blocks.
The filesystem on /dev/mapper/mds1_flakey is now 377837 (4k) blocks long.

oleg312-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/ost1_flakey to 591846 (4k) blocks.
The filesystem on /dev/mapper/ost1_flakey is now 589824 (4k) blocks long.

e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg312-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg312-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 20)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 24032 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48044 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48045 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48046 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48047 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48048 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48054 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] group 19 finished
[Thread 0] group 20 finished
[Thread 0] Pass 1: Memory used: 400k/0k (269k/132k), time:  0.00/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 207.51MB/s
[Thread 0] Scanned group range [0, 20), inodes 383
Pass 2: Checking directory structure
Pass 2: Memory used: 576k/0k (224k/353k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 123.02MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 576k/0k (224k/353k), time:  0.05/ 0.02/ 0.00
Pass 3A: Memory used: 576k/0k (224k/353k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 576k/0k (222k/355k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 3597.12MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 500k/0k (68k/433k), time:  0.02/ 0.01/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 500k/0k (67k/434k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 197.71MB/s

         382 inodes used (0.08%, out of 480000)
           5 non-contiguous files (1.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
      124660 blocks used (32.99%, out of 377837)
           0 bad blocks
           1 large file

         254 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         372 files
Memory used: 500k/0k (66k/435k), time:  0.08/ 0.04/ 0.00
I/O read: 1MB, write: 1MB, rate: 12.35MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg312-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg312-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 18)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 22562 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 22563 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] Pass 1: Memory used: 372k/0k (245k/128k), time:  0.02/ 0.01/ 0.00
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 233.20MB/s
[Thread 0] Scanned group range [0, 18), inodes 402
Pass 2: Checking directory structure
Pass 2: Memory used: 532k/0k (200k/333k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 142.05MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 532k/0k (200k/333k), time:  0.06/ 0.02/ 0.00
Pass 3A: Memory used: 532k/0k (200k/333k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 532k/0k (197k/336k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 4310.34MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 468k/0k (66k/403k), time:  0.02/ 0.01/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 468k/0k (65k/404k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 205.25MB/s

         402 inodes used (0.10%, out of 405072)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 396
       89417 blocks used (15.16%, out of 589824)
           0 bad blocks
           1 large file

         220 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         392 files
Memory used: 468k/0k (64k/405k), time:  0.08/ 0.04/ 0.00
I/O read: 2MB, write: 1MB, rate: 25.45MB/s
start mds service on oleg312-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=131072
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
check files after shrinking the MDT and OST filesystems
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-1 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-1 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-2 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-2 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-3 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-3 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-4 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-4 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-5 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-5 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-6 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-6 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-7 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-7 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-8 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-8 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-9 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-9 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-10 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-10 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-11 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-11 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-12 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-12 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-13 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-13 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-14 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-14 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-15 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-15 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-16 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-16 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-17 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-17 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-18 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-18 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-19 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-19 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-20 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-20 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-21 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-21 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-22 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-22 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-23 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-23 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-24 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-24 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-25 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-25 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-26 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-26 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-27 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-27 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-28 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-28 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-29 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-29 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-30 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-30 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-31 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-31 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-32 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-32 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-33 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-33 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-34 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-34 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-35 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-35 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-36 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-36 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-37 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-37 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-38 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-38 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-39 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-39 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-40 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-40 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-41 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-41 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-42 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-42 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-43 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-43 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-44 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-44 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-45 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-45 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-46 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-46 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-47 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-47 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-48 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-48 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-49 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-49 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-50 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-50 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-51 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-51 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-52 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-52 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-53 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-53 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-54 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-54 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-55 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-55 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-56 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-56 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-57 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-57 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-58 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-58 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-59 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-59 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-60 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-60 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-61 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-61 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-62 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-62 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-63 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-63 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-64 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-64 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-65 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-65 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-66 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-66 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-67 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-67 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-68 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-68 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-69 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-69 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-70 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-70 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-71 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-71 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-72 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-72 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-73 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-73 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-74 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-74 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-75 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-75 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-76 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-76 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-77 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-77 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-78 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-78 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-79 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-79 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-80 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-80 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-81 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-81 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-82 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-82 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-83 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-83 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-84 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-84 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-85 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-85 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-86 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-86 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-87 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-87 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-88 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-88 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-89 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-89 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-90 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-90 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-91 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-91 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-92 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-92 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-93 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-93 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-94 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-94 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-95 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-95 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-96 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-96 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-97 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-97 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-98 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-98 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-99 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-99 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-100 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-100 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-101 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-101 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-102 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-102 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-103 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-103 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-104 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-104 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-105 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-105 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-106 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-106 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-107 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-107 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-108 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-108 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-109 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-109 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-110 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-110 has size 1048576 OK
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 78 (242s)

== conf-sanity-1 test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:37:58 (1734259078)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
oleg312-server: 
oleg312-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg312-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 22
oleg312-server: 
oleg312-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg312-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 22
oleg312-server: 
oleg312-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg312-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 22
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 79 (79s)

== conf-sanity-1 test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 05:39:17 (1734259157)
mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem
mnt_opts -o loop
run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1
oleg312-server: oleg312-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no
oleg312-server: 
oleg312-server: llverfs: write /mnt/lustre-ost1/llverfs_dir00142/file000@0+1048576 short: 368640 written
oleg312-server: Timestamp: 1734259162
oleg312-server: dirs: 147, fs blocks: 37602
oleg312-server: write_done: /mnt/lustre-ost1/llverfs_dir00142/file000, current: 190.534 MB/s, overall: 190.534 MB/s, ETA: 0:00:00
oleg312-server: 
oleg312-server: read_done: /mnt/lustre-ost1/llverfs_dir00141/file000, current: 2756.75 MB/s, overall: 2756.75 MB/s, ETA: 0:00:00
oleg312-server: 
unmount the OST /dev/mapper/ost1_flakey
Stopping /mnt/lustre-ost1 (opts:) on oleg312-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg312-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: No space left on device
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 28
oleg312-server: error: set_param: param_path 'seq/cli-lustre': No such file or directory
oleg312-server: error: set_param: setting 'seq/cli-lustre'='OST0000-super.width=65536': No such file or directory
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 28
string err 
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 83 (81s)

== conf-sanity-1 test 86: Replacing mkfs.lustre -G option ========================================================== 05:40:39 (1734259239)
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps
params: --mgsnode=192.168.203.112@tcp --fsname=lustre --ost 			--index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-G 1024 -b 4096 -O flex_bg
 -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey 
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST0000
	kilobytes     200000
	options       -G 1024 -b 4096   -I 512 -q -O flex_bg,uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -G 1024 -b 4096   -I 512 -q -O flex_bg,uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 86 (65s)

== conf-sanity-1 test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 05:41:44 (1734259304)
Estimate: at most 351-byte space left in inode.
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
setup single mount lustre success
4 -rw-r--r-- 1 root root 67108865 Dec 15 05:42 /mnt/lustre-mds1/ROOT/f87.conf-sanity-1
Verified: at most 351-byte space left in inode.
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
PASS 87 (66s)

== conf-sanity-1 test 89: check tunefs --param and --erase-param{s} options ========================================================== 05:42:50 (1734259370)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
tunefs --param failover.node=192.0.2.254@tcp0
tunefs --param failover.node=192.0.2.255@tcp0
tunefs --erase-param failover.node
tunefs --erase-params
tunefs --param failover.node=192.0.2.254@tcp0 --erase-params
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 89 (73s)

== conf-sanity-1 test 93: register mulitple MDT at the same time ========================================================== 05:44:03 (1734259443)

 SKIP: conf-sanity-1 test_93 needs >= 3 MDTs
SKIP 93 (1s)

== conf-sanity-1 test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:44:04 (1734259444)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
setup single mount lustre success
lustre-MDT0000:
lustre-OST0000:
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 100 (72s)

== conf-sanity-1 test 104a: Make sure user defined options are reflected in mount ========================================================== 05:45:16 (1734259516)
mountfsopt: acl,user_xattr
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
Starting mds1: -o localrecov,noacl  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
Starting mds2: -o localrecov,noacl  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
setfacl: /mnt/lustre: Operation not supported
PASS 104a (51s)

== conf-sanity-1 test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:46:08 (1734259568)
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
oleg312-server: rmmod: ERROR: Module lustre is in use
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
touch: cannot touch '/tmp/d105.conf-sanity-1/f105.conf-sanity-1': Read-only file system
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 105 (105s)

== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:47:52 (1734259672)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992         274       79718   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896         368       32528   2% /mnt/lustre[OST:0]

filesystem_summary:        33057         529       32528   2% /mnt/lustre

seq.cli-lustre-OST0000-super.width=0x1ffffff
fail_loc=0x1312
fail_val=5
 - open/close 2336 (time 1734259731.34 total 10.00 last 233.55)
 - open/close 4477 (time 1734259741.34 total 20.00 last 214.04)
 - open/close 6662 (time 1734259751.34 total 30.01 last 218.48)
 - open/close 8782 (time 1734259761.34 total 40.01 last 211.99)
 - open/close 10000 (time 1734259766.31 total 44.97 last 245.29)
 - open/close 12492 (time 1734259776.31 total 54.97 last 249.14)
 - open/close 15069 (time 1734259786.31 total 64.98 last 257.65)
 - open/close 17630 (time 1734259796.31 total 74.98 last 256.03)
 - open/close 19782 (time 1734259806.31 total 84.98 last 215.20)
 - open/close 22092 (time 1734259816.31 total 94.98 last 230.97)
 - open/close 24634 (time 1734259826.32 total 104.98 last 254.17)
 - open/close 27396 (time 1734259836.32 total 114.98 last 276.18)
 - open/close 30000 (time 1734259845.73 total 124.39 last 276.65)
open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device
total: 32560 open/close in 133.46 seconds: 243.97 ops/second
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992       32835       47157  42% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896       32896           0 100% /mnt/lustre[OST:0]

filesystem_summary:        33090       33090           0 100% /mnt/lustre

 - /unlink 4993 (time 1734259865.88 total 10.00 last 499.28)
 - /unlink 9181 (time 1734259875.88 total 20.00 last 418.77)
 - /unlink 13293 (time 1734259885.88 total 30.00 last 411.15)
 - /unlink 17291 (time 1734259895.89 total 40.01 last 399.66)
 - /unlink 20000 (time 1734259903.04 total 47.16 last 378.73)
 - /unlink 24401 (time 1734259913.04 total 57.16 last 440.08)
 - /unlink 29241 (time 1734259923.04 total 67.16 last 483.95)
unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory
total: 32560 /unlink in 74.07 seconds: 439.59 ops/second
Waiting for MDT destroys to complete
oleg312-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg312-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
 - open/close 2548 (time 1734259948.47 total 10.00 last 254.73)
 - open/close 5091 (time 1734259958.48 total 20.00 last 254.27)
 - open/close 7554 (time 1734259968.48 total 30.01 last 246.22)
 - open/close 10000 (time 1734259977.86 total 39.38 last 260.86)
 - open/close 12640 (time 1734259987.86 total 49.38 last 264.00)
 - open/close 15481 (time 1734259997.86 total 59.38 last 284.10)
 - open/close 18820 (time 1734260007.86 total 69.39 last 333.84)
 - open/close 20000 (time 1734260011.43 total 72.96 last 330.48)
 - open/close 22943 (time 1734260021.43 total 82.96 last 294.27)
 - open/close 25848 (time 1734260031.43 total 92.96 last 290.44)
 - open/close 28765 (time 1734260041.43 total 102.96 last 291.64)
 - open/close 30000 (time 1734260046.21 total 107.74 last 258.46)
open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device
total: 32560 open/close in 115.79 seconds: 281.20 ops/second
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992       32835       47157  42% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896       32896           0 100% /mnt/lustre[OST:0]

filesystem_summary:        33090       33090           0 100% /mnt/lustre

 - /unlink 7408 (time 1734260065.23 total 10.00 last 740.73)
 - /unlink 10000 (time 1734260069.09 total 13.86 last 671.52)
 - /unlink 15929 (time 1734260079.09 total 23.86 last 592.85)
 - /unlink 20000 (time 1734260085.86 total 30.63 last 601.46)
 - /unlink 25784 (time 1734260095.86 total 40.63 last 578.34)
 - /unlink 30000 (time 1734260103.11 total 47.88 last 581.82)
unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory
total: 32560 /unlink in 52.00 seconds: 626.13 ops/second
Waiting for MDT destroys to complete
oleg312-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg312-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
 - open/close 2933 (time 1734260133.68 total 10.00 last 293.28)
 - open/close 5897 (time 1734260143.68 total 20.00 last 296.37)
 - open/close 9195 (time 1734260153.68 total 30.00 last 329.75)
 - open/close 10000 (time 1734260155.98 total 32.30 last 349.92)
 - open/close 13645 (time 1734260165.99 total 42.31 last 364.45)
 - open/close 16902 (time 1734260175.99 total 52.31 last 325.65)
 - open/close 20000 (time 1734260185.30 total 61.62 last 332.60)
 - open/close 23455 (time 1734260195.30 total 71.62 last 345.43)
 - open/close 26448 (time 1734260205.30 total 81.62 last 299.27)
 - open/close 29530 (time 1734260215.31 total 91.63 last 308.13)
 - open/close 32503 (time 1734260225.31 total 101.63 last 297.28)
open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device
total: 32560 open/close in 101.81 seconds: 319.81 ops/second
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992       32835       47157  42% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896       32896           0 100% /mnt/lustre[OST:0]

filesystem_summary:        33090       33090           0 100% /mnt/lustre

 - /unlink 5485 (time 1734260236.51 total 10.00 last 548.45)
 - /unlink 10000 (time 1734260245.00 total 18.49 last 532.11)
 - /unlink 15812 (time 1734260255.00 total 28.49 last 581.19)
 - /unlink 20000 (time 1734260261.53 total 35.01 last 641.68)
 - /unlink 25817 (time 1734260271.53 total 45.01 last 581.66)
 - /unlink 30000 (time 1734260279.53 total 53.02 last 522.76)
unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory
total: 32560 /unlink in 57.74 seconds: 563.87 ops/second
Waiting for MDT destroys to complete
oleg312-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg312-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
 - open/close 2609 (time 1734260300.94 total 10.00 last 260.84)
 - open/close 5406 (time 1734260310.94 total 20.00 last 279.66)
 - open/close 8130 (time 1734260320.94 total 30.01 last 272.33)
 - open/close 10000 (time 1734260326.96 total 36.03 last 310.54)
 - open/close 12921 (time 1734260336.97 total 46.03 last 292.06)
 - open/close 15912 (time 1734260346.97 total 56.03 last 299.07)
 - open/close 19048 (time 1734260356.97 total 66.03 last 313.59)
 - open/close 20000 (time 1734260360.28 total 69.34 last 287.56)
 - open/close 22795 (time 1734260370.28 total 79.34 last 279.44)
 - open/close 25485 (time 1734260380.28 total 89.34 last 269.00)
 - open/close 28111 (time 1734260390.28 total 99.35 last 262.52)
 - open/close 30000 (time 1734260397.80 total 106.87 last 251.22)
 - open/close 32433 (time 1734260407.80 total 116.87 last 243.25)
open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device
total: 32560 open/close in 117.49 seconds: 277.13 ops/second
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992       32835       47157  42% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896       32896           0 100% /mnt/lustre[OST:0]

filesystem_summary:        33090       33090           0 100% /mnt/lustre

 - /unlink 5000 (time 1734260419.61 total 10.00 last 499.97)
 - /unlink 9990 (time 1734260429.61 total 20.00 last 498.94)
 - /unlink 14725 (time 1734260439.61 total 30.00 last 473.46)
 - /unlink 19369 (time 1734260449.62 total 40.00 last 464.34)
 - /unlink 24561 (time 1734260459.62 total 50.01 last 519.13)
 - /unlink 29856 (time 1734260469.62 total 60.01 last 529.48)
unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory
total: 32560 /unlink in 64.89 seconds: 501.81 ops/second
Waiting for MDT destroys to complete
oleg312-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
oleg312-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
 - open/close 3068 (time 1734260494.09 total 10.00 last 306.73)
 - open/close 6166 (time 1734260504.09 total 20.00 last 309.77)
 - open/close 9154 (time 1734260514.09 total 30.00 last 298.76)
 - open/close 10000 (time 1734260516.83 total 32.75 last 308.48)
 - open/close 13018 (time 1734260526.83 total 42.75 last 301.79)
 - open/close 16378 (time 1734260536.84 total 52.75 last 335.96)
 - open/close 19096 (time 1734260546.84 total 62.75 last 271.80)
 - open/close 20000 (time 1734260550.41 total 66.33 last 252.72)
 - open/close 22880 (time 1734260560.42 total 76.33 last 287.87)
 - open/close 25963 (time 1734260570.42 total 86.33 last 308.29)
 - open/close 29174 (time 1734260580.42 total 96.33 last 321.05)
 - open/close 30000 (time 1734260582.62 total 98.54 last 374.66)
open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device
total: 32560 open/close in 105.83 seconds: 307.67 ops/second
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992       32835       47157  42% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896       32896           0 100% /mnt/lustre[OST:0]

filesystem_summary:        33090       33090           0 100% /mnt/lustre

 - /unlink 5801 (time 1734260600.89 total 10.00 last 580.09)
 - /unlink 10000 (time 1734260608.64 total 17.75 last 541.65)
 - /unlink 16203 (time 1734260618.64 total 27.75 last 620.23)
 - /unlink 20000 (time 1734260624.64 total 33.75 last 633.16)
 - /unlink 25514 (time 1734260634.64 total 43.75 last 551.27)
 - /unlink 30000 (time 1734260642.31 total 51.42 last 584.73)
unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory
total: 32560 /unlink in 55.28 seconds: 588.98 ops/second
Waiting for MDT destroys to complete
oleg312-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
oleg312-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
fail_loc=0
fail_val=0
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 106 (1003s)

== conf-sanity-1 test 109a: test lctl clear_conf fsname == 06:04:35 (1734260675)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Setting lustre-MDT0000.mdd.atime_diff from 60 to 62
Waiting 90s for '62'
Updated after 8s: want '62' got '62'
Setting lustre-MDT0000.mdd.atime_diff from 62 to 63
Waiting 90s for '63'
Updated after 6s: want '63' got '63'
Setting lustre.llite.max_read_ahead_mb from 256 to 32
Waiting 90s for '32'
Updated after 3s: want '32' got '32'
Setting lustre.llite.max_read_ahead_mb from 32 to 64
Waiting 90s for '64'
Updated after 6s: want '64' got '64'
oleg312-server: Pool lustre.pool1 created
oleg312-server: OST lustre-OST0000_UUID added to pool lustre.pool1
oleg312-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg312-server: OST lustre-OST0000_UUID added to pool lustre.pool1
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
start mds service on oleg312-server
Starting mds1: -o localrecov -o nosvc  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
Start /dev/mapper/mds1_flakey without service
Started lustre-MDT0000
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Destroy the created pools: pool1
lustre.pool1
oleg312-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg312-server: Pool lustre.pool1 destroyed
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 109a (175s)

== conf-sanity-1 test 109b: test lctl clear_conf one config ========================================================== 06:07:30 (1734260850)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Setting lustre-MDT0000.mdd.atime_diff from 60 to 62
Waiting 90s for '62'
Updated after 8s: want '62' got '62'
Setting lustre-MDT0000.mdd.atime_diff from 62 to 63
Waiting 90s for '63'
Updated after 7s: want '63' got '63'
Setting lustre.llite.max_read_ahead_mb from 256 to 32
Waiting 90s for '32'
Updated after 2s: want '32' got '32'
Setting lustre.llite.max_read_ahead_mb from 32 to 64
Waiting 90s for '64'
Updated after 6s: want '64' got '64'
oleg312-server: Pool lustre.pool1 created
Waiting 90s for ''
oleg312-server: OST lustre-OST0000_UUID added to pool lustre.pool1
oleg312-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg312-server: OST lustre-OST0000_UUID added to pool lustre.pool1
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
start mds service on oleg312-server
Starting mds1: -o localrecov -o nosvc  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
Start /dev/mapper/mds1_flakey without service
Started lustre-MDT0000
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Destroy the created pools: pool1
lustre.pool1
oleg312-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg312-server: Pool lustre.pool1 destroyed
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 109b (162s)

== conf-sanity-1 test 116: big size MDT support ========== 06:10:12 (1734261012)
/usr/sbin/mkfs.xfs
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
meta-data=/tmp/f116.conf-sanity-1-mdt0 isize=512    agcount=4, agsize=67108864 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=268435456, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=131072, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: not found
formatting backing filesystem ldiskfs on /dev/loop1
	target name   lustre:MDT0000
	kilobytes     18253611008
	options       -i 16777216 -b 4096  -J size=4096 -I 1024 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,64bit,^resize_inode,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,lazy_itable_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -i 16777216 -b 4096  -J size=4096 -I 1024 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,64bit,^resize_inode,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,lazy_itable_init,packed_meta_blocks -F /dev/loop1 18253611008k
Writing CONFIGS/mountdata
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 116 (65s)

== conf-sanity-1 test 122a: Check OST sequence update ==== 06:11:17 (1734261077)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
fail_loc=0x00001e0
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
fail_loc=0
total: 1000 open/close in 1.80 seconds: 554.94 ops/second
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 122a (60s)

== conf-sanity-1 test 123ah: del_ost cancels config log entries correctly ========================================================== 06:12:17 (1734261137)
Checking servers environments
Checking clients oleg312-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Starting client oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Started clients oleg312-client.virtnet: 
192.168.203.112@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800a96f1800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800a96f1800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 6s: want 'procname_uid' got 'procname_uid'
disable quota as required
del_ost: dry run for target lustre-OST0000
config_log: lustre-MDT0000
[DRY RUN] cancel catalog 'lustre-MDT0000:29':"- { index: 29, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-MDT0000:28':"- { index: 28, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.203.112@tcp }"
[DRY RUN] cancel catalog 'lustre-MDT0000:27':"- { index: 27, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-client
[DRY RUN] cancel catalog 'lustre-client:34':"- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-client:33':"- { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.203.112@tcp }"
[DRY RUN] cancel catalog 'lustre-client:32':"- { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-MDT0001
[DRY RUN] cancel catalog 'lustre-MDT0001:32':"- { index: 32, event: add_osc, device: lustre-MDT0001-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-MDT0001:31':"- { index: 31, event: setup, device: lustre-OST0000-osc-MDT0001, UUID: lustre-OST0000_UUID, node: 192.168.203.112@tcp }"
[DRY RUN] cancel catalog 'lustre-MDT0001:30':"- { index: 30, event: attach, device: lustre-OST0000-osc-MDT0001, type: osc, UUID: lustre-MDT0001-mdtlov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-MDT0000
cancel catalog lustre-MDT0000 log_idx 29: done
cancel catalog lustre-MDT0000 log_idx 28: done
cancel catalog lustre-MDT0000 log_idx 27: done
del_ost: cancelled 3 catalog entries
config_log: lustre-client
cancel catalog lustre-client log_idx 34: done
cancel catalog lustre-client log_idx 33: done
cancel catalog lustre-client log_idx 32: done
del_ost: cancelled 3 catalog entries
config_log: lustre-MDT0001
cancel catalog lustre-MDT0001 log_idx 32: done
cancel catalog lustre-MDT0001 log_idx 31: done
cancel catalog lustre-MDT0001 log_idx 30: done
del_ost: cancelled 3 catalog entries
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
oleg312-server: rmmod: ERROR: Module lustre is in use
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
fail_loc=0
PASS 123ah (120s)

== conf-sanity-1 test 132: hsm_actions processed after failover ========================================================== 06:14:17 (1734261257)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mdt.hsm_control=enabled

Writing CONFIGS/mountdata
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
PASS 132 (23s)

== conf-sanity-1 test 133: stripe QOS: free space balance in a pool ========================================================== 06:14:40 (1734261280)

 SKIP: conf-sanity-1 test_133 needs >= 4 OSTs
SKIP 133 (1s)

== conf-sanity-1 test 135: check the behavior when changelog is wrapped around ========================================================== 06:14:41 (1734261281)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
oleg312-client: fail_loc=0x1312
oleg312-client: fail_val=5
oleg312-server: fail_loc=0x1312
oleg312-server: fail_val=5
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity-1
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
total: 4500 open/close in 8.72 seconds: 515.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #12998
total: 4500 /unlink in 4.53 seconds: 992.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
total: 4500 open/close in 8.70 seconds: 517.41 ops/second
total: 4500 /unlink in 4.75 seconds: 947.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
total: 4500 open/close in 8.66 seconds: 519.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
total: 4500 /unlink in 4.69 seconds: 958.76 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 10973: 24132 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
PASS 135 (106s)

== conf-sanity-1 test 136: don't panic with bad obdecho setup ========================================================== 06:16:27 (1734261387)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
oleg312-server: oleg312-server.virtnet: executing load_module obdecho/obdecho
oleg312-server: error: setup: Invalid argument
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 22
oleg312-server: error: test_mkdir: No such file or directory
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping client oleg312-client.virtnet /mnt/lustre opts:
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 136 (110s)

== conf-sanity-1 test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:18:17 (1734261497)

 SKIP: conf-sanity-1 test_137 needs >= 3 MDTs
SKIP 137 (1s)

== conf-sanity-1 test 140: remove_updatelog script actions ========================================================== 06:18:18 (1734261498)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
striped dir -i0 -c2 -H crush /mnt/lustre/d140.conf-sanity-1
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Dry run was requested, no changes will be applied
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 0 1
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
Processing MDT1 llog catalog [0x240000400:0x1:0x0] ...
remove_updatelog: /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] is too small.
> /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0]
Dry run was requested, no changes will be applied
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 1 0
Processing MDT1 llog catalog [0x240000400:0x1:0x0] ...
remove_updatelog: /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] is too small.
> /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0]
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 0
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping client oleg312-client.virtnet /mnt/lustre opts:
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 140 (229s)

== conf-sanity-1 test 151a: damaged local config doesn't prevent mounting ========================================================== 06:22:07 (1734261727)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg312-server
stop mds service on oleg312-server
stop mds service on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
Damage ost1 local config log
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
start ost1 service on oleg312-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg312-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Invalid argument
oleg312-server: This may have multiple causes.
oleg312-server: Are the mount options correct?
oleg312-server: Check the syslog for more info.
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 22
oleg312-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg312-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 22
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 151a (133s)

== conf-sanity-1 test 151b: -ENOSPC doesn't affect mount ========================================================== 06:24:20 (1734261860)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg312-server
stop mds service on oleg312-server
stop mds service on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
start mds service on oleg312-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
fail_loc=0x131e
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
PASS 151b (95s)

== conf-sanity-1 test 153a: bypass invalid NIDs quickly == 06:25:55 (1734261955)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
waiting for mount ...
waiting for mount ...
          "192.168.203.112@tcp": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 5 }
          "192.168.252.112@tcp": { connects: 1, replied: 0, uptodate: discovering, sec_ago: 10 }
          "10.252.252.113@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
          "192.168.203.112@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
setup single mount lustre success
umount lustre on /mnt/lustre.....
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 153a (194s)

== conf-sanity-1 test 153b: added IPv6 NID support ======= 06:29:09 (1734262149)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Checking servers environments
Checking clients oleg312-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Starting client oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Started clients oleg312-client.virtnet: 
192.168.203.112@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800a9e80000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800a9e80000.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 4s: want 'procname_uid' got 'procname_uid'
disable quota as required
PASS 153b (94s)

== conf-sanity-1 test 153c: don't stuck on unreached NID ========================================================== 06:30:43 (1734262243)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
start mds service on oleg312-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg312-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
          "192.168.252.112@tcp": { connects: 3, replied: 0, uptodate: discovering, sec_ago: 0 }
          "192.168.252.113@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 10 }
          "192.168.252.115@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 5 }
connection attempts: 7
Waiting for mount to fail
mount.lustre: mount 192.168.252.112@tcp:192.168.252.113@tcp:192.168.252.115@tcp:/lustre at /mnt/lustre failed: Input/output error
Is the MGS running?
umount lustre on /mnt/lustre.....
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
PASS 153c (354s)

== conf-sanity-1 test 154: expand .. on rename after MDT backup restore ========================================================== 06:36:37 (1734262597)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
Checking servers environments
Checking clients oleg312-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Starting client oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Started clients oleg312-client.virtnet: 
192.168.203.112@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff88012aaa2800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff88012aaa2800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 3s: want 'procname_uid' got 'procname_uid'
disable quota as required
striped dir -i0 -c1 -H all_char /mnt/lustre/d154.conf-sanity-1
striped dir -i0 -c1 -H crush2 /mnt/lustre/d154.conf-sanity-1.tgt
striped dir -i0 -c1 -H crush2 /mnt/lustre/d154.conf-sanity-1.dx_dir
striped dir -i0 -c1 -H all_char /mnt/lustre/d154.conf-sanity-1.dirent
striped dir -i0 -c1 -H crush2 /mnt/lustre/d154.conf-sanity-1.empty
total: 128 open/close in 0.53 seconds: 243.28 ops/second
total: 2 open/close in 0.03 seconds: 79.53 ops/second
==target parent FID: fid:[0x200000402:0x1:0x0]==
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping client oleg312-client.virtnet /mnt/lustre opts:
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
==debugfs before backup==
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 53385   Type: directory    Mode:  0755   Flags: 0x1000
Generation: 3887566429    Version: 0x00000001:0000010c
User:     0   Group:     0   Project:     0   Size: 16384
File ACL: 0
Links: 2   Blockcount: 32
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebf8a:00000000 -- Sun Dec 15 06:37:46 2024
 atime: 0x675ebf88:00000000 -- Sun Dec 15 06:37:44 2024
 mtime: 0x675ebf8a:00000000 -- Sun Dec 15 06:37:46 2024
crtime: 0x675ebf89:1f78a400 -- Sun Dec 15 06:37:45 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x3:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  user.job (5) = "lfs.0"
  linkea: idx=0 parent=[0x200000402:0x1:0x0] name='d154.conf-sanity-1.dx_dir'
BLOCKS:
(0):24781, (1-3):24784-24786
TOTAL: 4

oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  53385   40755 (2)      0      0   16384 15-Dec-2024 06:37 .
  53383   40755 (18)      0      0    4096 15-Dec-2024 06:37 fid:[0x200000402:0x1:0x0] ..
    171  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0xc:0x0] f154.conf-sanity-1.longfilename6
    172  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0xd:0x0] f154.conf-sanity-1.longfilename7
    174  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0xf:0x0] f154.conf-sanity-1.longfilename9
    176  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x11:0x0] f154.conf-sanity-1.longfilename11
    179  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x14:0x0] f154.conf-sanity-1.longfilename14
    182  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x17:0x0] f154.conf-sanity-1.longfilename17
    183  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x18:0x0] f154.conf-sanity-1.longfilename18
    186  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x1b:0x0] f154.conf-sanity-1.longfilename21
    192  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x21:0x0] f154.conf-sanity-1.longfilename27
    197  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x26:0x0] f154.conf-sanity-1.longfilename32
    199  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x28:0x0] f154.conf-sanity-1.longfilename34
    200  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x29:0x0] f154.conf-sanity-1.longfilename35
    207  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x30:0x0] f154.conf-sanity-1.longfilename42
    217  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x3a:0x0] f154.conf-sanity-1.longfilename52
    221  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x3e:0x0] f154.conf-sanity-1.longfilename56
    224  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x41:0x0] f154.conf-sanity-1.longfilename59
    230  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x47:0x0] f154.conf-sanity-1.longfilename65
    233  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x4a:0x0] f154.conf-sanity-1.longfilename68
    234  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x4b:0x0] f154.conf-sanity-1.longfilename69
    235  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x4c:0x0] f154.conf-sanity-1.longfilename70
    236  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x4d:0x0] f154.conf-sanity-1.longfilename71
    237  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x4e:0x0] f154.conf-sanity-1.longfilename72
    246  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x57:0x0] f154.conf-sanity-1.longfilename81
    252  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x5d:0x0] f154.conf-sanity-1.longfilename87
    255  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x60:0x0] f154.conf-sanity-1.longfilename90
    257  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x62:0x0] f154.conf-sanity-1.longfilename92
    262  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x67:0x0] f154.conf-sanity-1.longfilename97
    265  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x6a:0x0] f154.conf-sanity-1.longfilename100
    267  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x6c:0x0] f154.conf-sanity-1.longfilename102
    268  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x6d:0x0] f154.conf-sanity-1.longfilename103
    269  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x6e:0x0] f154.conf-sanity-1.longfilename104
    271  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x70:0x0] f154.conf-sanity-1.longfilename106
    273  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x72:0x0] f154.conf-sanity-1.longfilename108
    282  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x7b:0x0] f154.conf-sanity-1.longfilename117
    286  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x7f:0x0] f154.conf-sanity-1.longfilename121
    288  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x81:0x0] f154.conf-sanity-1.longfilename123
    290  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x83:0x0] f154.conf-sanity-1.longfilename125
    191  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x20:0x0] f154.conf-sanity-1.longfilename26
    166  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x7:0x0] f154.conf-sanity-1.longfilename1
    167  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x8:0x0] f154.conf-sanity-1.longfilename2
    223  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x40:0x0] f154.conf-sanity-1.longfilename58
    213  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x36:0x0] f154.conf-sanity-1.longfilename48
    226  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x43:0x0] f154.conf-sanity-1.longfilename61
    215  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x38:0x0] f154.conf-sanity-1.longfilename50
    178  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x13:0x0] f154.conf-sanity-1.longfilename13
    228  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x45:0x0] f154.conf-sanity-1.longfilename63
    206  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x2f:0x0] f154.conf-sanity-1.longfilename41
    187  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x1c:0x0] f154.conf-sanity-1.longfilename22
    193  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x22:0x0] f154.conf-sanity-1.longfilename28
    173  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0xe:0x0] f154.conf-sanity-1.longfilename8
    169  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0xa:0x0] f154.conf-sanity-1.longfilename4
    218  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x3b:0x0] f154.conf-sanity-1.longfilename53
    204  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x2d:0x0] f154.conf-sanity-1.longfilename39
    231  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x48:0x0] f154.conf-sanity-1.longfilename66
    205  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x2e:0x0] f154.conf-sanity-1.longfilename40
    201  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x2a:0x0] f154.conf-sanity-1.longfilename36
    196  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x25:0x0] f154.conf-sanity-1.longfilename31
    202  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x2b:0x0] f154.conf-sanity-1.longfilename37
    165  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x6:0x0] f154.conf-sanity-1.longfilename0
    229  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x46:0x0] f154.conf-sanity-1.longfilename64
    222  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x3f:0x0] f154.conf-sanity-1.longfilename57
    185  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x1a:0x0] f154.conf-sanity-1.longfilename20
    170  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0xb:0x0] f154.conf-sanity-1.longfilename5
    209  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x32:0x0] f154.conf-sanity-1.longfilename44
    225  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x42:0x0] f154.conf-sanity-1.longfilename60
    195  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x24:0x0] f154.conf-sanity-1.longfilename30
    177  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x12:0x0] f154.conf-sanity-1.longfilename12
    194  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x23:0x0] f154.conf-sanity-1.longfilename29
    216  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x39:0x0] f154.conf-sanity-1.longfilename51
    208  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x31:0x0] f154.conf-sanity-1.longfilename43
    238  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x4f:0x0] f154.conf-sanity-1.longfilename73
    239  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x50:0x0] f154.conf-sanity-1.longfilename74
    241  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x52:0x0] f154.conf-sanity-1.longfilename76
    243  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x54:0x0] f154.conf-sanity-1.longfilename78
    244  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x55:0x0] f154.conf-sanity-1.longfilename79
    245  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x56:0x0] f154.conf-sanity-1.longfilename80
    247  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x58:0x0] f154.conf-sanity-1.longfilename82
    248  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x59:0x0] f154.conf-sanity-1.longfilename83
    250  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x5b:0x0] f154.conf-sanity-1.longfilename85
    256  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x61:0x0] f154.conf-sanity-1.longfilename91
    258  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x63:0x0] f154.conf-sanity-1.longfilename93
    260  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x65:0x0] f154.conf-sanity-1.longfilename95
    272  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x71:0x0] f154.conf-sanity-1.longfilename107
    275  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x74:0x0] f154.conf-sanity-1.longfilename110
    276  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x75:0x0] f154.conf-sanity-1.longfilename111
    277  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x76:0x0] f154.conf-sanity-1.longfilename112
    279  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x78:0x0] f154.conf-sanity-1.longfilename114
    280  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x79:0x0] f154.conf-sanity-1.longfilename115
    281  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x7a:0x0] f154.conf-sanity-1.longfilename116
    283  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x7c:0x0] f154.conf-sanity-1.longfilename118
    287  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x80:0x0] f154.conf-sanity-1.longfilename122
    289  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x82:0x0] f154.conf-sanity-1.longfilename124
    291  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x84:0x0] f154.conf-sanity-1.longfilename126
    292  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x85:0x0] f154.conf-sanity-1.longfilename127
    240  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x51:0x0] f154.conf-sanity-1.longfilename75
    270  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x6f:0x0] f154.conf-sanity-1.longfilename105
    232  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x49:0x0] f154.conf-sanity-1.longfilename67
    189  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x1e:0x0] f154.conf-sanity-1.longfilename24
    190  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x1f:0x0] f154.conf-sanity-1.longfilename25
    214  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x37:0x0] f154.conf-sanity-1.longfilename49
    220  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x3d:0x0] f154.conf-sanity-1.longfilename55
    278  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x77:0x0] f154.conf-sanity-1.longfilename113
    266  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x6b:0x0] f154.conf-sanity-1.longfilename101
    181  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x16:0x0] f154.conf-sanity-1.longfilename16
    263  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x68:0x0] f154.conf-sanity-1.longfilename98
    242  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x53:0x0] f154.conf-sanity-1.longfilename77
    251  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x5c:0x0] f154.conf-sanity-1.longfilename86
    249  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x5a:0x0] f154.conf-sanity-1.longfilename84
    184  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x19:0x0] f154.conf-sanity-1.longfilename19
    264  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x69:0x0] f154.conf-sanity-1.longfilename99
    254  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x5f:0x0] f154.conf-sanity-1.longfilename89
    211  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x34:0x0] f154.conf-sanity-1.longfilename46
    227  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x44:0x0] f154.conf-sanity-1.longfilename62
    168  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x9:0x0] f154.conf-sanity-1.longfilename3
    259  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x64:0x0] f154.conf-sanity-1.longfilename94
    284  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x7d:0x0] f154.conf-sanity-1.longfilename119
    210  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x33:0x0] f154.conf-sanity-1.longfilename45
    175  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x10:0x0] f154.conf-sanity-1.longfilename10
    198  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x27:0x0] f154.conf-sanity-1.longfilename33
    212  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x35:0x0] f154.conf-sanity-1.longfilename47
    253  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x5e:0x0] f154.conf-sanity-1.longfilename88
    188  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x1d:0x0] f154.conf-sanity-1.longfilename23
    180  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x15:0x0] f154.conf-sanity-1.longfilename15
    261  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x66:0x0] f154.conf-sanity-1.longfilename96
    274  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x73:0x0] f154.conf-sanity-1.longfilename109
    219  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x3c:0x0] f154.conf-sanity-1.longfilename54
    285  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x7e:0x0] f154.conf-sanity-1.longfilename120
    203  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x2c:0x0] f154.conf-sanity-1.longfilename38

oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 53386   Type: directory    Mode:  0755   Flags: 0x0
Generation: 3887566430    Version: 0x00000001:0000010d
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebf8a:00000000 -- Sun Dec 15 06:37:46 2024
 atime: 0x675ebf88:00000000 -- Sun Dec 15 06:37:44 2024
 mtime: 0x675ebf8a:00000000 -- Sun Dec 15 06:37:46 2024
crtime: 0x675ebf89:2b646600 -- Sun Dec 15 06:37:45 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x4:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  user.job (5) = "lfs.0"
  linkea: idx=0 parent=[0x200000402:0x1:0x0] name='d154.conf-sanity-1.dirent'
BLOCKS:
(0):24782
TOTAL: 1

oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  53386   40755 (2)      0      0    4096 15-Dec-2024 06:37 .
  53383   40755 (18)      0      0    4096 15-Dec-2024 06:37 fid:[0x200000402:0x1:0x0] ..
    293  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x86:0x0] f154.conf-sanity-1.longfilename0
    294  100644 (17)      0      0       0 15-Dec-2024 06:37 fid:[0x200000402:0x87:0x0] f154.conf-sanity-1.longfilename1

oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 53387   Type: directory    Mode:  0755   Flags: 0x0
Generation: 3887566431    Version: 0x00000001:0000010e
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebf8a:00000000 -- Sun Dec 15 06:37:46 2024
 atime: 0x675ebf88:00000000 -- Sun Dec 15 06:37:44 2024
 mtime: 0x675ebf88:00000000 -- Sun Dec 15 06:37:44 2024
crtime: 0x675ebf89:378d3100 -- Sun Dec 15 06:37:45 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x5:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  user.job (5) = "lfs.0"
  linkea: idx=0 parent=[0x200000402:0x1:0x0] name='d154.conf-sanity-1.empty'
BLOCKS:
(0):24783
TOTAL: 1

oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  53387   40755 (2)      0      0    4096 15-Dec-2024 06:37 .
  53383   40755 (18)      0      0    4096 15-Dec-2024 06:37 fid:[0x200000402:0x1:0x0] ..

file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
Checking servers environments
Checking clients oleg312-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Starting client oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
Started clients oleg312-client.virtnet: 
192.168.203.112@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8801309ab000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8801309ab000.idle_timeout=debug
disable quota as required
==pre-rename parent/.. inodes==
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:37 /mnt/lustre/d154.conf-sanity-1
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:37 /mnt/lustre/d154.conf-sanity-1/d154.conf-sanity-1.dx_dir/..
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:37 /mnt/lustre/d154.conf-sanity-1
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:37 /mnt/lustre/d154.conf-sanity-1/d154.conf-sanity-1.dirent/..
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:37 /mnt/lustre/d154.conf-sanity-1
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:37 /mnt/lustre/d154.conf-sanity-1/d154.conf-sanity-1.empty/..
==post-rename parent/.. inodes==
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:39 /mnt/lustre/d154.conf-sanity-1.tgt
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:39 /mnt/lustre/d154.conf-sanity-1.tgt/d154.conf-sanity-1.dx_dir/..
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:39 /mnt/lustre/d154.conf-sanity-1.tgt
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:39 /mnt/lustre/d154.conf-sanity-1.tgt/d154.conf-sanity-1.dirent/..
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:39 /mnt/lustre/d154.conf-sanity-1.tgt
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:39 /mnt/lustre/d154.conf-sanity-1.tgt/d154.conf-sanity-1.empty/..
==target parent FID: fid:[0x200000402:0x2:0x0]==
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:)
Stopping client oleg312-client.virtnet /mnt/lustre opts:
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg312-server
==post-rename .dx_dir should have 'fid:[0x200000402:0x2:0x0] ..'==
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 26672   Type: directory    Mode:  0755   Flags: 0x1000
Generation: 2231959116    Version: 0x00000002:00000001
User:     0   Group:     0   Project:     0   Size: 12288
File ACL: 0
Links: 2   Blockcount: 24
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebfda:00000000 -- Sun Dec 15 06:39:06 2024
 atime: 0x675ebfb3:38074300 -- Sun Dec 15 06:38:27 2024
 mtime: 0x675ebf8a:00000000 -- Sun Dec 15 06:37:46 2024
crtime: 0x675ebfb3:38074300 -- Sun Dec 15 06:38:27 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x3:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  linkea: idx=0 parent=[0x200000402:0x2:0x0] name='d154.conf-sanity-1.dx_dir'
BLOCKS:
(0-2):24160-24162
TOTAL: 3

oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  26672   40755 (2)      0      0   12288 15-Dec-2024 06:37 .
  26669   40755 (18)      0      0    4096 15-Dec-2024 06:39 fid:[0x200000402:0x2:0x0] ..
     58  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename85
     59  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename26
     60  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename1
     61  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename2
     62  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename110
     63  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename83
     67  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename50
     73  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename80
     75  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename73
     76  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename4
     78  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename53
     80  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename66
     81  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename40
     82  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename79
     83  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename36
     84  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename112
     87  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename116
     88  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename37
     90  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename74
     92  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename64
     95  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename78
     98  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename20
     99  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename114
    100  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename5
    101  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename115
    102  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename82
    104  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename44
    105  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename60
    106  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename30
    107  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename124
    110  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename29
    111  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename51
    115  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename69
    121  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename42
    122  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename71
    128  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename6
    129  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename21
    132  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename14
    133  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename125
    135  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename104
    136  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename100
    138  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename32
    139  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename90
    143  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename68
    145  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename35
    149  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename103
    150  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename52
    151  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename75
    161  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename98
    162  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename77
    166  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename99
    169  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename62
    172  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename119
    174  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename10
    176  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename47
    177  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename88
    179  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename15
    180  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename96
    183  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename120
    184  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename38
     72  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename28
    140  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename56
     93  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename118
     64  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename58
    117  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename72
     96  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename91
     79  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename39
    116  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename34
    126  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename70
    118  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename65
    134  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename17
    109  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename76
     86  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename122
    123  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename59
    131  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename108
     77  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename127
     69  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename63
    120  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename92
    124  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename102
     65  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename48
    113  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename95
    130  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename9
     74  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename8
    108  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename12
    148  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename27
    125  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename7
    103  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename111
    114  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename11
     71  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename22
    141  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename106
     89  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename126
     68  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename13
    112  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename43
     85  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename31
    147  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename121
     66  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename61
    119  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename123
    137  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename117
    146  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename97
     91  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename0
    144  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename18
     70  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename41
     57  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename107
    127  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename87
    142  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename81
     94  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename93
     97  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename57
    152  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename105
    153  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename67
    154  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename24
    155  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename25
    156  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename49
    157  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename55
    158  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename113
    159  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename101
    160  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename16
    163  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename86
    164  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename84
    165  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename19
    167  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename89
    168  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename46
    170  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename3
    171  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename94
    173  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename45
    175  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename33
    178  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename23
    181  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename109
    182  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename54

==post-rename .dirent should have 'fid:[0x200000402:0x2:0x0] ..'==
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 26671   Type: directory    Mode:  0755   Flags: 0x0
Generation: 2231959113    Version: 0x00000002:00000002
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebfda:00000000 -- Sun Dec 15 06:39:06 2024
 atime: 0x675ebfb3:37ca3a00 -- Sun Dec 15 06:38:27 2024
 mtime: 0x675ebf8a:00000000 -- Sun Dec 15 06:37:46 2024
crtime: 0x675ebfb3:37ca3a00 -- Sun Dec 15 06:38:27 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x4:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  linkea: idx=0 parent=[0x200000402:0x2:0x0] name='d154.conf-sanity-1.dirent'
BLOCKS:
(0):24207
TOTAL: 1

oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  26671   40755 (2)      0      0    4096 15-Dec-2024 06:37 .
  26669   40755 (18)      0      0    4096 15-Dec-2024 06:39 fid:[0x200000402:0x2:0x0] ..
     55  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename1
     56  100644 (1)      0      0       0 15-Dec-2024 06:37 f154.conf-sanity-1.longfilename0

==post-rename .empty should have 'fid:[0x200000402:0x2:0x0] ..'==
oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 26673   Type: directory    Mode:  0755   Flags: 0x0
Generation: 2231959245    Version: 0x00000002:00000003
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebfda:00000000 -- Sun Dec 15 06:39:06 2024
 atime: 0x675ebfb3:3eb43f00 -- Sun Dec 15 06:38:27 2024
 mtime: 0x675ebf88:00000000 -- Sun Dec 15 06:37:44 2024
crtime: 0x675ebfb3:3eb43f00 -- Sun Dec 15 06:38:27 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x5:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  linkea: idx=0 parent=[0x200000402:0x2:0x0] name='d154.conf-sanity-1.empty'
BLOCKS:
(0):24163
TOTAL: 1

oleg312-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg312-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  26673   40755 (2)      0      0    4096 15-Dec-2024 06:37 .
  26669   40755 (18)      0      0    4096 15-Dec-2024 06:39 fid:[0x200000402:0x2:0x0] ..

e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8
oleg312-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg312-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 3)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 209 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 291 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 292 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 294 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] Pass 1: Memory used: 276k/0k (144k/133k), time:  0.00/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 382.12MB/s
[Thread 0] Scanned group range [0, 3), inodes 414
Pass 2: Checking directory structure
Pass 2: Memory used: 276k/0k (98k/179k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 442.28MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 276k/0k (98k/179k), time:  0.01/ 0.00/ 0.00
Pass 3: Memory used: 276k/0k (96k/181k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 276k/0k (68k/209k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 276k/0k (67k/210k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 358.81MB/s

         412 inodes used (0.52%, out of 79992)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
       24569 blocks used (49.14%, out of 50000)
           0 bad blocks
           1 large file

         280 regular files
         122 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         402 files
Memory used: 276k/0k (66k/211k), time:  0.02/ 0.01/ 0.00
I/O read: 1MB, write: 0MB, rate: 65.23MB/s
PASS 154 (213s)
Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f)
oleg312-server: oleg312-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
== conf-sanity-1 test complete, duration 7732 sec ======== 06:40:28 (1734262828)
=== conf-sanity-1: start cleanup 06:40:29 (1734262829) ===
=== conf-sanity-1: finish cleanup 06:40:30 (1734262830) ===