-----============= acceptance-small: conf-sanity-1 ============----- Sun Dec 15 04:32:09 EST 2024
mgs: CentOS Linux release 7.9.2009 (Core)
MGS_OS_VERSION_ID=7
MGS_OS_ID=centos
MGS_OS_VERSION_CODE=117440512
MGS_OS_ID_LIKE=rhel fedora centos
mds1: CentOS Linux release 7.9.2009 (Core)
MDS1_OS_ID_LIKE=rhel fedora centos
MDS1_OS_ID=centos
MDS1_OS_VERSION_ID=7
MDS1_OS_VERSION_CODE=117440512
ost1: CentOS Linux release 7.9.2009 (Core)
OST1_OS_VERSION_CODE=117440512
OST1_OS_VERSION_ID=7
OST1_OS_ID_LIKE=rhel fedora centos
OST1_OS_ID=centos
client: CentOS Linux release 7.9.2009 (Core)
CLIENT_OS_ID=centos
CLIENT_OS_ID_LIKE=rhel fedora centos
CLIENT_OS_VERSION_ID=7
CLIENT_OS_VERSION_CODE=117440512
oleg422-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity-1.*ex: No such file or directory
excepting tests: 
skipping tests SLOW=no: 106
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg422-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server

== conf-sanity-1 test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 04:33:47 (1734255227)
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
Remove mds config log
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
start ost1 service on oleg422-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: No such file or directory
oleg422-server: Is the MGS specification correct?
oleg422-server: Is the filesystem name correct?
oleg422-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
Start of /dev/mapper/mds1_flakey on mds1 failed 2
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 17 (186s)

== conf-sanity-1 test 18: check mkfs creates large journals ========================================================== 04:36:53 (1734255413)
use device /dev/mapper/mds1_flakey with MIN=2000000
mount mds with large journal...
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
mount lustre system...
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
check journal size...
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Success: mkfs creates large journals. Size: 78M
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 18 (137s)

== conf-sanity-1 test 21c: start mds between two osts, stop mds last ========================================================== 04:39:10 (1734255550)
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost2 service on oleg422-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop ost2 service on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

PASS 21c (106s)

== conf-sanity-1 test 29: permanently remove an OST ====== 04:40:56 (1734255656)
start ost2 service on oleg422-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0001
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Setting lustre-OST0001.osc.active from 1 to 0
Waiting 90s for '0'
Updated after 5s: want '0' got '0'
Live client success: got lustre-OST0001_UUID	FULL	DEACTIVATED
osc.lustre-OST0001-osc-MDT0000.active updated after 0 sec (got 0)
osc.lustre-OST0001-osc-MDT0001.active updated after 0 sec (got 0)
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
New client success: got 'lustre-OST0001_UUID	NEW	DEACTIVATED'
Setting lustre-OST0001.osc.active from 0 to 1
Waiting 90s for '1'
Updated after 6s: want '1' got '1'
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 29 (165s)

== conf-sanity-1 test 35a: Reconnect to the last active server first ========================================================== 04:43:41 (1734255821)
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
debug=ha
Set up a fake failnode for the MDS
Wait for RECONNECT_INTERVAL seconds (10s)
conf-sanity.sh test_35a 2024-12-15 4h44m25s
Stopping the MDT: lustre-MDT0000
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Restarting the MDT: lustre-MDT0000
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
Wait for df (8998) ... 
done
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout
Debug log: 101 lines, 101 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 35a (155s)

== conf-sanity-1 test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:46:16 (1734255976)
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
debug=ha
conf-sanity.sh test_35b 2024-12-15 4h46m51s
Set up a fake failnode for the MDS
at_max=0
at_max=0
Injecting EBUSY on MDS
fail_loc=0x80000136
mdc.lustre-MDT0000-mdc-ffff880130175800.stats=clear
mdc.lustre-MDT0001-mdc-ffff880130175800.stats=clear
Creating a test file and stat it
  File: '/mnt/lustre/d35b.conf-sanity-1/f35b.conf-sanity-1'
  Size: 0         	Blocks: 0          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d	Inode: 144115238826934274  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:47:36.000000000 -0500
Modify: 2024-12-15 04:47:36.000000000 -0500
Change: 2024-12-15 04:47:36.000000000 -0500
 Birth: -
Stop injecting EBUSY on MDS
fail_loc=0
done
at_max=600
at_max=600
Debug log: 76 lines, 76 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 35b (175s)

== conf-sanity-1 test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:49:11 (1734256151)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov -o nosvc -n  /dev/mapper/mds1_flakey /mnt/lustre-mds1
nomtab: 1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
Start /dev/mapper/mds1_flakey without service
Commit the device label on /dev/mapper/mds1_flakey
Started lustre:MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg422-server
Starting mds1: -o localrecov -o nomgs,force  /dev/mapper/mds1_flakey /mnt/lustre-mds1
force: 1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
blah blah
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
PASS 41b (87s)

== conf-sanity-1 test 46a: handle ost additional - wide striped file ========================================================== 04:50:38 (1734256238)
Testing with 2 OSTs
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800aa248800.ost_server_uuid 50
osc.lustre-OST0001-osc-ffff8800aa248800.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre2.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2
/mnt/lustre2
stripe_count:  -1 stripe_size:   4194304 pattern:       raid0 stripe_offset: -1

/mnt/lustre2/f41b.conf-sanity-1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	             2	          0x2	   0x280000400

/mnt/lustre2/widestripe
lmm_stripe_count:  2
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 1
	obdidx		 objid		 objid		 group
	     1	             2	          0x2	   0x2c0000401
	     0	            34	         0x22	   0x280000400

  File: '/mnt/lustre/widestripe'
  Size: 3         	Blocks: 1          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115238826934273  Links: 1
Access: (0674/-rw-rwxr--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:51:08.000000000 -0500
Modify: 2024-12-15 04:51:08.000000000 -0500
Change: 2024-12-15 04:51:09.000000000 -0500
 Birth: -
umount lustre on /mnt/lustre2.....
Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:)
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
stop ost1 service on oleg422-server
stop mds service on oleg422-server
stop mds service on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 46a (119s)

== conf-sanity-1 test 47a: server restart does not lose lru_resize ========================================================== 04:52:37 (1734256357)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012d008800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff88012d008800.lru_size=100
ldlm.namespaces.lustre-OST0000-osc-ffff88012d008800.lru_size=100
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012d008800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff88012d008800.lru_size=100
Failing ost1 on oleg422-server
Stopping /mnt/lustre-ost1 (opts:) on oleg422-server
04:53:17 (1734256397) shut down
facet: ost1 facet_host: oleg422-server facet_failover_host: oleg422-server
Failover ost1 to oleg422-server
mount facets: ost1
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
04:53:32 (1734256412) targets are mounted
04:53:32 (1734256412) facet_failover done
Failing mds1 on oleg422-server
Stopping /mnt/lustre-mds1 (opts:) on oleg422-server
04:53:34 (1734256414) shut down
facet: mds1 facet_host: oleg422-server facet_failover_host: oleg422-server
Failover mds1 to oleg422-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
04:53:48 (1734256428) targets are mounted
04:53:48 (1734256428) facet_failover done
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012d008800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff88012d008800.lru_size=100
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 47a (96s)

== conf-sanity-1 test 47b: client restart does not lose lru_resize ========================================================== 04:54:13 (1734256453)
Failing mds1 on oleg422-server
04:54:14 (1734256454) shut down
facet: mds1 facet_host: oleg422-server facet_failover_host: oleg422-server
Failover mds1 to oleg422-server
mount facets: mds1
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
04:54:37 (1734256477) targets are mounted
04:54:37 (1734256477) facet_failover done
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 95
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 95
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
ln: failed to create symbolic link '/home/green/git/lustre-release/lustre/utils/lctl': File exists

 SKIP: conf-sanity-1 test_47b Can not create symlink /usr/sbin/lctl
SKIP 47b (67s)

== conf-sanity-1 test 48: too many acls on file ========== 04:55:20 (1734256520)
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg422-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
mount.lustre: according to /etc/mtab 192.168.204.122@tcp:/lustre is already mounted on /mnt/lustre
setup single mount lustre success
debug=0
debug=0
/mnt/lustre
stripe_count:  -1 stripe_size:   4194304 pattern:       raid0 stripe_offset: -1

/mnt/lustre/widestripe
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	             4	          0x4	   0x280000401

It is expected to hold at least 4500 ACL entries
  File: '/mnt/lustre/widestripe'
  Size: 3         	Blocks: 8          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115205289279491  Links: 1
Access: (0664/-rw-rw-r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-15 04:55:26.000000000 -0500
Modify: 2024-12-15 04:55:26.000000000 -0500
Change: 2024-12-15 04:56:32.000000000 -0500
 Birth: -
getfacl: Removing leading '/' from absolute path names
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg422-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 48 (156s)

== conf-sanity-1 test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:57:56 (1734256676)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
check ldlm_timout...
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 49a (55s)

== conf-sanity-1 test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:58:51 (1734256731)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 49b (76s)

== conf-sanity-1 test 50c: lazystatfs one server down ==== 05:00:07 (1734256807)
start mds service on oleg422-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg422-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800abab9000.lazystatfs=1
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
oleg422-server: oleg422-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7631
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        95248        1680       84912   2% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        95248        1532       85060   2% /mnt/lustre[MDT:1]
lustre-OST0001_UUID       142216        1528      126688   2% /mnt/lustre[OST:1]

filesystem_summary:       142216        1528      126688   2% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (30992) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19

PASS 50c (81s)

== conf-sanity-1 test 50d: lazystatfs client/server conn race ========================================================== 05:01:28 (1734256888)
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg422-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0001
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800b5848000.lazystatfs=1
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7631
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        95248        1680       84912   2% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        95248        1532       85060   2% /mnt/lustre[MDT:1]
lustre-OST0001_UUID       142216        1528      126688   2% /mnt/lustre[OST:1]

filesystem_summary:       142216        1528      126688   2% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity-framework.sh: line 241: kill: (1398) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19

PASS 50d (71s)

== conf-sanity-1 test 50f: normal statfs one server in down ========================================================== 05:02:39 (1734256959)
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
start ost2 service on oleg422-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0001
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
oleg422-server: oleg422-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
llite.lustre-ffff88012d00b000.lazystatfs=0
multiop /mnt/lustre v_f
TMPPIPE=/tmp/multiop_open_wait_pipe.7631
start ost2 service on oleg422-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0001
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19

PASS 50f (120s)

== conf-sanity-1 test 50g: deactivated OST should not cause panic ========================================================== 05:04:39 (1734257079)
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
start ost2 service on oleg422-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0001
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800aa931800.ost_server_uuid 50
osc.lustre-OST0001-osc-ffff8800aa931800.ost_server_uuid in FULL state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Filesystem                  1K-blocks  Used Available Use% Mounted on
192.168.204.122@tcp:/lustre    142216  1524    126692   2% /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 sys.ldlm_timeout=19

PASS 50g (69s)

== conf-sanity-1 test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:05:48 (1734257148)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
fail_loc=0x142
start ost2 service on oleg422-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 51 (168s)

== conf-sanity-1 test 54a: test llverdev and partial verify of device ========================================================== 05:08:36 (1734257316)
oleg422-server: oleg422-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p
oleg422-server: llverdev: /dev/mapper/ost1_flakey is 4294967296 bytes (4 GB) in size
oleg422-server: timestamp: 1734257318 chunksize: 1048576 size: 4294967296
oleg422-server: write offset:        4194304kB   inf MB/s            
oleg422-server: read offset:        4194304kB   inf MB/s            
oleg422-server: llverdev: data verified successfully
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 54a (57s)

== conf-sanity-1 test 55: check lov_objid size =========== 05:09:33 (1734257373)

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre:OST03ff
Index:      1023
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST03ff
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST03ff -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: mount.lustre: mount /dev/mapper/mds2_flakey at /mnt/lustre-mds2 failed: No such file or directory
oleg422-server: Is the MGS specification correct?
oleg422-server: Is the filesystem name correct?
oleg422-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
Start of /dev/mapper/mds2_flakey on mds2 failed 2
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST03ff-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST03ff
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03ff-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping client oleg422-client.virtnet /mnt/lustre opts:
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: mount.lustre: mount /dev/mapper/mds2_flakey at /mnt/lustre-mds2 failed: No such file or directory
oleg422-server: Is the MGS specification correct?
oleg422-server: Is the filesystem name correct?
oleg422-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
Start of /dev/mapper/mds2_flakey on mds2 failed 2
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST03ff-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST03ff
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03ff-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
checking size of lov_objid for ost index 1023
/home/green/git/lustre-release/lustre/tests/conf-sanity-1.sh: line 841: lov_objid_size: command not found
/home/green/git/lustre-release/lustre/tests/conf-sanity-1.sh: line 841: [: 8192: unary operator expected
ok, lov_objid size is correct: 8192
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping client oleg422-client.virtnet /mnt/lustre opts:
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre:OST0800
Index:      2048
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST0800
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0800 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: mount.lustre: mount /dev/mapper/mds2_flakey at /mnt/lustre-mds2 failed: No such file or directory
oleg422-server: Is the MGS specification correct?
oleg422-server: Is the filesystem name correct?
oleg422-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
Start of /dev/mapper/mds2_flakey on mds2 failed 2
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0800-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0800
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0800-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping client oleg422-client.virtnet /mnt/lustre opts:
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: mount.lustre: mount /dev/mapper/mds2_flakey at /mnt/lustre-mds2 failed: No such file or directory
oleg422-server: Is the MGS specification correct?
oleg422-server: Is the filesystem name correct?
oleg422-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
Start of /dev/mapper/mds2_flakey on mds2 failed 2
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0800-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0800
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0800-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
checking size of lov_objid for ost index 2048
/home/green/git/lustre-release/lustre/tests/conf-sanity-1.sh: line 841: lov_objid_size: command not found
/home/green/git/lustre-release/lustre/tests/conf-sanity-1.sh: line 841: [: 16392: unary operator expected
ok, lov_objid size is correct: 16392
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping client oleg422-client.virtnet /mnt/lustre opts:
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 55 (172s)

== conf-sanity-1 test 56a: check big OST indexes and out-of-index-order start ========================================================== 05:12:25 (1734257545)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey

   Permanent disk data:
Target:     lustre:OST2710
Index:      10000
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST2710
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST2710 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre:OST03e8
Index:      1000
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost2_flakey
	target name   lustre:OST03e8
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST03e8 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost2_flakey 200000k
Writing CONFIGS/mountdata
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST2710-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST2710
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg422-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST03e8-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST03e8
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
ok
OBDS:
1000: lustre-OST03e8_UUID ACTIVE
10000: lustre-OST2710_UUID ACTIVE
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.00542728 s, 755 kB/s
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping client oleg422-client.virtnet /mnt/lustre opts:
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 56a (157s)

== conf-sanity-1 test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:15:02 (1734257702)

 SKIP: conf-sanity-1 test_56b needs >= 3 MDTs
SKIP 56b (2s)

== conf-sanity-1 test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:15:04 (1734257704)
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x142
              (OST update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 failover.node=192.168.204.122@tcp

Writing CONFIGS/mountdata
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg422-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Cannot assign requested address
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 99
oleg422-server: error: set_param: param_path 'seq/cli-lustre': No such file or directory
oleg422-server: error: set_param: setting 'seq/cli-lustre'='OST0000-super.width=65536': No such file or directory
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 99
umount lustre on /mnt/lustre.....
stop ost1 service on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 57a (40s)

== conf-sanity-1 test 57b: initial registration from servicenode should not fail ========================================================== 05:15:44 (1734257744)
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 failover.node=192.168.204.122@tcp


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 failover.node=192.168.204.122@tcp

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x142
              (OST update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x142
              (OST update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 failover.node=192.168.204.122@tcp


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1142
              (OST update writeconf no_primnode )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20  failover.node=192.168.204.122@tcp:192.168.204.122@tcp

Writing CONFIGS/mountdata
start mds service on oleg422-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 57b (60s)

== conf-sanity-1 test 59: writeconf mount option ========= 05:16:45 (1734257805)
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
original ost count: 1 (expect > 0)
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
after mdt writeconf count: 0 (expect 0)
OST start without writeconf should fail:
oleg422-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: No such file or directory
oleg422-server: Is the MGS specification correct?
oleg422-server: Is the filesystem name correct?
oleg422-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg422-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
OST start with writeconf should succeed:
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
after ost writeconf count: 1 (expect 1)
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
after ost2 writeconf count: 2 (expect 2)
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1002
              (OST no_primnode )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20  failover.node=192.168.204.122@tcp:192.168.204.122@tcp


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1102
              (OST writeconf no_primnode )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20  failover.node=192.168.204.122@tcp:192.168.204.122@tcp

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 59 (160s)

== conf-sanity-1 test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 05:19:24 (1734257964)

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options          -I 1024 -i 2560 -q -O ^uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000    -I 1024 -i 2560 -q -O ^uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre:MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds2_flakey
	target name   lustre:MDT0001
	kilobytes     200000
	options          -I 1024 -i 2560 -q -O ^uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0001    -I 1024 -i 2560 -q -O ^uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E stride=64,lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/mapper/mds2_flakey 200000k
Writing CONFIGS/mountdata
oleg422-server: dumpe2fs 1.46.2.wc5 (26-Mar-2022)
stop mds service on oleg422-server
stop mds service on oleg422-server
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 60a (73s)

== conf-sanity-1 test 62: start with disabled journal ==== 05:20:37 (1734258037)
disable journal for mds
tune2fs 1.46.2.wc5 (26-Mar-2022)
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Invalid argument
oleg422-server: This may have multiple causes.
oleg422-server: Are the mount options correct?
oleg422-server: Check the syslog for more info.
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22
Start of /dev/mapper/mds1_flakey on mds1 failed 22
disable journal for ost
tune2fs 1.46.2.wc5 (26-Mar-2022)
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg422-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Invalid argument
oleg422-server: This may have multiple causes.
oleg422-server: Are the mount options correct?
oleg422-server: Check the syslog for more info.
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22
oleg422-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg422-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 22
umount lustre on /mnt/lustre.....
stop ost1 service on oleg422-server
stop mds service on oleg422-server
stop mds service on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 62 (76s)

== conf-sanity-1 test 64: check lfs df --lazy ============ 05:21:53 (1734258113)
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg422-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
stop ost2 service on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
/home/green/git/lustre-release/lustre/utils/lfs df
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        95248        1668       84924   2% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        95248        1532       85060   2% /mnt/lustre[MDT:1]
lustre-OST0000_UUID       142216        1524      126692   2% /mnt/lustre[OST:0]

filesystem_summary:       142216        1524      126692   2% /mnt/lustre

umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

oleg422-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device
oleg422-server: Is the ldiskfs module available?
oleg422-server: 
oleg422-server: tunefs.lustre FATAL: failed to write local files
oleg422-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

tunefs failed, reformatting instead
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 64 (149s)

== conf-sanity-1 test 66: replace nids =================== 05:24:22 (1734258262)
client=34615898 MDS=34615898 OSS=34615898
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Setting lustre-OST0000.osc.active from 1 to 0
Waiting 90s for '0'
Updated after 3s: want '0' got '0'
replace_nids should fail if MDS, OSTs and clients are UP
oleg422-server: error: replace_nids: Operation now in progress
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 115
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
replace_nids should fail if MDS and OSTs are UP
oleg422-server: error: replace_nids: Operation now in progress
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 115
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
replace_nids should fail if MDS is UP
oleg422-server: error: replace_nids: Operation now in progress
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 115
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
start mds service on oleg422-server
Starting mds1: -o localrecov -o nosvc  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
Start /dev/mapper/mds1_flakey without service
Started lustre-MDT0000
command should accept two parameters
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
correct device name should be passed
oleg422-server: error: replace_nids: No such device or address
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 6
wrong nids list should not destroy the system
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
replace OST nid
command should accept two parameters
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
wrong nids list should not destroy the system
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 4
replace primary NIDs for device (clients/servers must be unmounted)
usage: replace_nids <DEVICE> <NID1>[,NID2,NID3:NID4,NID5:NID6]
set NIDs with failover
replace MDS nid
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Setting lustre-OST0000.osc.active from 0 to 1
Waiting 90s for '1'
Waiting 80s for '1'
Waiting 70s for '1'
Updated after 21s: want '1' got '1'
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 66 (189s)

== conf-sanity-1 test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 05:27:31 (1734258451)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST0000
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid

   Permanent disk data:
Target:     lustre:MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds2_flakey
	target name   lustre:MDT0001
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0001 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds2_flakey 200000k
Writing CONFIGS/mountdata
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-server: oleg422-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50
oleg422-server: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50
oleg422-server: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
umount lustre on /mnt/lustre.....
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 70e (89s)

== conf-sanity-1 test 75: The order of --index should be irrelevant ========================================================== 05:29:00 (1734258540)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre-MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x42
              (OST update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre-OST0000
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre-MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x42
              (OST update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre-OST0000
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 75 (67s)

== conf-sanity-1 test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:30:07 (1734258607)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format ost1: /dev/mapper/ost1_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=131072
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
create test files
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        83240        1616       73832   3% /mnt/lustre[MDT:0]
lustre-OST0000_UUID       124712        1388      110724   2% /mnt/lustre[OST:0]

filesystem_summary:       124712        1388      110724   2% /mnt/lustre

UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        72000         272       71728   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID        45008         302       44706   1% /mnt/lustre[OST:0]

filesystem_summary:        44978         272       44706   1% /mnt/lustre

1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0431938 s, 24.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0425821 s, 24.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0458821 s, 22.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0435439 s, 24.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0400761 s, 26.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0432984 s, 24.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0409521 s, 25.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0678738 s, 15.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0429087 s, 24.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0493335 s, 21.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0453613 s, 23.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0478225 s, 21.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0439971 s, 23.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0507861 s, 20.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0441045 s, 23.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0443201 s, 23.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0455217 s, 23.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0409936 s, 25.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0433163 s, 24.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.042767 s, 24.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0429033 s, 24.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0429848 s, 24.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0451279 s, 23.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0441716 s, 23.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0438468 s, 23.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0439223 s, 23.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0464163 s, 22.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0451694 s, 23.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.045221 s, 23.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0446098 s, 23.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0447427 s, 23.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0444972 s, 23.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0468151 s, 22.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0433093 s, 24.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0421702 s, 24.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0422423 s, 24.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0463771 s, 22.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0491633 s, 21.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0468507 s, 22.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0452273 s, 23.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0432534 s, 24.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0488765 s, 21.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0452625 s, 23.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0453688 s, 23.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.044575 s, 23.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.044875 s, 23.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0453434 s, 23.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0670989 s, 15.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0457448 s, 22.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0461052 s, 22.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0444263 s, 23.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0460082 s, 22.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0446722 s, 23.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0426957 s, 24.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0437896 s, 23.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0446607 s, 23.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0488929 s, 21.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0472549 s, 22.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0437355 s, 24.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0431431 s, 24.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0463107 s, 22.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0427015 s, 24.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0477994 s, 21.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0443836 s, 23.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0473312 s, 22.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0456555 s, 23.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0468493 s, 22.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0475551 s, 22.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0690364 s, 15.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0452159 s, 23.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0434681 s, 24.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0441734 s, 23.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0457629 s, 22.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0440414 s, 23.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0450425 s, 23.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0445657 s, 23.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0506993 s, 20.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.047016 s, 22.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0440822 s, 23.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0469983 s, 22.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0451657 s, 23.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0694522 s, 15.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0475386 s, 22.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0447998 s, 23.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0450123 s, 23.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0675466 s, 15.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.476131 s, 2.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0444677 s, 23.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.046275 s, 22.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0662311 s, 15.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0452443 s, 23.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0466027 s, 22.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.04133 s, 25.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0528257 s, 19.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0494597 s, 21.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0639759 s, 16.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0431803 s, 24.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0427376 s, 24.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.074026 s, 14.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0907516 s, 11.6 MB/s
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg422-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg422-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 3)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 24054 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24055 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24056 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24057 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24058 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24060 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48009 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] Pass 1: Memory used: 272k/0k (140k/133k), time:  0.00/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 263.85MB/s
[Thread 0] Scanned group range [0, 3), inodes 373
Pass 2: Checking directory structure
Pass 2: Memory used: 272k/0k (95k/178k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 290.87MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 272k/0k (95k/178k), time:  0.02/ 0.01/ 0.01
Pass 3A: Memory used: 272k/0k (95k/178k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 272k/0k (93k/180k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 4651.16MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 272k/0k (67k/206k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 272k/0k (67k/206k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 298.33MB/s

         372 inodes used (0.52%, out of 72000)
           4 non-contiguous files (1.1%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
       22546 blocks used (50.10%, out of 45000)
           0 bad blocks
           1 large file

         244 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         362 files
Memory used: 272k/0k (66k/207k), time:  0.03/ 0.01/ 0.01
I/O read: 1MB, write: 1MB, rate: 37.86MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg422-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg422-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 2)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] group 2 finished
[Thread 0] Pass 1: Memory used: 264k/0k (132k/133k), time:  0.01/ 0.00/ 0.01
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 502.92MB/s
[Thread 0] Scanned group range [0, 2), inodes 398
Pass 2: Checking directory structure
Pass 2: Memory used: 264k/0k (87k/178k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 211.01MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 264k/0k (92k/173k), time:  0.02/ 0.00/ 0.01
Pass 3A: Memory used: 264k/0k (92k/173k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 264k/0k (84k/181k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 4424.78MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 264k/0k (65k/200k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 264k/0k (64k/201k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 358.81MB/s

         398 inodes used (0.88%, out of 45008)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 392
       37721 blocks used (83.82%, out of 45000)
           0 bad blocks
           1 large file

         216 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         388 files
Memory used: 264k/0k (64k/201k), time:  0.03/ 0.01/ 0.01
I/O read: 2MB, write: 1MB, rate: 71.39MB/s
oleg422-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/mds1_flakey to 640000 (4k) blocks.
The filesystem on /dev/mapper/mds1_flakey is now 640000 (4k) blocks long.

oleg422-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/ost1_flakey to 1048576 (4k) blocks.
The filesystem on /dev/mapper/ost1_flakey is now 1048576 (4k) blocks long.

e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg422-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg422-server: Use max possible thread num: 2 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 16)
[Thread 1] Scan group range [16, 33)
[Thread 0] jumping to group 0
[Thread 1] jumping to group 16
[Thread 1] group 17 finished
[Thread 1] group 18 finished
[Thread 1] group 19 finished
[Thread 1] group 20 finished
[Thread 1] group 21 finished
[Thread 1] group 22 finished
[Thread 1] group 23 finished
[Thread 1] group 24 finished
[Thread 1] group 25 finished
[Thread 1] group 26 finished
[Thread 1] group 27 finished
[Thread 1] group 28 finished
[Thread 1] group 29 finished
[Thread 1] group 30 finished
[Thread 1] group 31 finished
[Thread 1] group 32 finished
[Thread 1] group 33 finished
[Thread 1] Pass 1: Memory used: 632k/0k (380k/253k), time:  0.00/ 0.00/ 0.00
[Thread 1] Pass 1: I/O read: 1MB, write: 0MB, rate: 2298.85MB/s
[Thread 1] Scanned group range [16, 33), inodes 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 24054 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24055 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24056 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24057 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24058 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24060 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48009 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] Pass 1: Memory used: 688k/0k (354k/335k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 193.31MB/s
[Thread 0] Scanned group range [0, 16), inodes 373
Pass 2: Checking directory structure
Pass 2: Memory used: 632k/0k (199k/434k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 215.15MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 632k/0k (200k/433k), time:  0.04/ 0.03/ 0.01
Pass 3A: Memory used: 632k/0k (200k/433k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 632k/0k (198k/435k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 3521.13MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 632k/0k (71k/562k), time:  0.02/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 632k/0k (70k/563k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 266.74MB/s

         372 inodes used (0.05%, out of 792000)
           4 non-contiguous files (1.1%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
      202726 blocks used (31.68%, out of 640000)
           0 bad blocks
           1 large file

         244 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         362 files
Memory used: 632k/0k (69k/564k), time:  0.07/ 0.06/ 0.01
I/O read: 1MB, write: 1MB, rate: 14.15MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg422-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg422-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 32)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] group 19 finished
[Thread 0] group 20 finished
[Thread 0] group 21 finished
[Thread 0] group 22 finished
[Thread 0] group 23 finished
[Thread 0] group 24 finished
[Thread 0] group 25 finished
[Thread 0] group 26 finished
[Thread 0] group 27 finished
[Thread 0] group 28 finished
[Thread 0] group 29 finished
[Thread 0] group 30 finished
[Thread 0] group 31 finished
[Thread 0] group 32 finished
[Thread 0] Pass 1: Memory used: 472k/0k (344k/129k), time:  0.01/ 0.00/ 0.01
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 410.14MB/s
[Thread 0] Scanned group range [0, 32), inodes 398
Pass 2: Checking directory structure
Pass 2: Memory used: 684k/0k (299k/386k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 142.33MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 684k/0k (299k/386k), time:  0.04/ 0.02/ 0.01
Pass 3A: Memory used: 684k/0k (299k/386k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 684k/0k (296k/389k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 6802.72MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 568k/0k (66k/503k), time:  0.02/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 568k/0k (65k/504k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 197.28MB/s

         398 inodes used (0.06%, out of 720128)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 392
      128315 blocks used (12.24%, out of 1048576)
           0 bad blocks
           1 large file

         216 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         388 files
Memory used: 568k/0k (64k/505k), time:  0.06/ 0.04/ 0.01
I/O read: 2MB, write: 1MB, rate: 31.06MB/s
start mds service on oleg422-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=131072
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
check files after expanding the MDT and OST filesystems
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-1 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-1 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-2 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-2 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-3 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-3 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-4 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-4 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-5 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-5 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-6 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-6 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-7 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-7 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-8 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-8 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-9 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-9 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-10 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-10 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-11 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-11 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-12 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-12 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-13 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-13 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-14 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-14 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-15 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-15 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-16 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-16 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-17 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-17 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-18 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-18 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-19 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-19 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-20 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-20 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-21 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-21 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-22 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-22 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-23 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-23 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-24 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-24 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-25 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-25 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-26 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-26 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-27 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-27 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-28 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-28 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-29 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-29 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-30 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-30 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-31 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-31 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-32 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-32 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-33 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-33 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-34 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-34 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-35 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-35 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-36 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-36 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-37 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-37 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-38 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-38 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-39 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-39 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-40 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-40 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-41 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-41 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-42 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-42 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-43 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-43 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-44 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-44 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-45 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-45 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-46 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-46 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-47 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-47 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-48 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-48 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-49 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-49 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-50 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-50 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-51 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-51 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-52 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-52 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-53 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-53 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-54 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-54 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-55 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-55 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-56 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-56 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-57 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-57 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-58 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-58 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-59 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-59 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-60 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-60 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-61 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-61 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-62 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-62 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-63 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-63 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-64 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-64 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-65 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-65 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-66 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-66 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-67 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-67 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-68 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-68 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-69 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-69 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-70 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-70 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-71 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-71 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-72 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-72 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-73 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-73 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-74 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-74 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-75 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-75 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-76 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-76 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-77 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-77 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-78 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-78 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-79 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-79 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-80 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-80 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-81 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-81 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-82 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-82 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-83 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-83 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-84 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-84 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-85 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-85 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-86 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-86 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-87 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-87 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-88 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-88 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-89 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-89 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-90 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-90 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-91 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-91 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-92 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-92 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-93 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-93 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-94 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-94 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-95 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-95 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-96 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-96 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-97 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-97 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-98 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-98 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-99 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-99 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-100 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-100 has size 1048576 OK
create more files after expanding the MDT and OST filesystems
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0481752 s, 21.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0420866 s, 24.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0462546 s, 22.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0446543 s, 23.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0460618 s, 22.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0438785 s, 23.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0481406 s, 21.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0812172 s, 12.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0501318 s, 20.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0462223 s, 22.7 MB/s
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg422-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg422-server: Use max possible thread num: 2 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 16)
[Thread 1] Scan group range [16, 33)
[Thread 0] jumping to group 0
[Thread 1] jumping to group 16
[Thread 1] group 17 finished
[Thread 1] group 18 finished
[Thread 1] group 19 finished
[Thread 1] group 20 finished
[Thread 1] group 21 finished
[Thread 1] group 22 finished
[Thread 1] group 23 finished
[Thread 1] group 24 finished
[Thread 1] group 25 finished
[Thread 1] group 26 finished
[Thread 1] group 27 finished
[Thread 1] group 28 finished
[Thread 1] group 29 finished
[Thread 1] group 30 finished
[Thread 1] group 31 finished
[Thread 1] group 32 finished
[Thread 1] group 33 finished
[Thread 1] Pass 1: Memory used: 632k/0k (381k/252k), time:  0.00/ 0.00/ 0.00
[Thread 1] Pass 1: I/O read: 1MB, write: 0MB, rate: 3663.00MB/s
[Thread 1] Scanned group range [16, 33), inodes 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 24054 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24055 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24056 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24057 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24058 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24060 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48009 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] Pass 1: Memory used: 688k/0k (355k/334k), time:  0.01/ 0.01/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 122.50MB/s
[Thread 0] Scanned group range [0, 16), inodes 383
Pass 2: Checking directory structure
Pass 2: Memory used: 632k/0k (199k/434k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 165.95MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 632k/0k (200k/433k), time:  0.05/ 0.04/ 0.00
Pass 3A: Memory used: 632k/0k (200k/433k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 632k/0k (198k/435k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 3436.43MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 632k/0k (71k/562k), time:  0.02/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 632k/0k (70k/563k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 215.38MB/s

         382 inodes used (0.05%, out of 792000)
           4 non-contiguous files (1.0%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
      202726 blocks used (31.68%, out of 640000)
           0 bad blocks
           1 large file

         254 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         372 files
Memory used: 632k/0k (69k/564k), time:  0.08/ 0.07/ 0.00
I/O read: 1MB, write: 1MB, rate: 12.94MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg422-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg422-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 32)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] group 19 finished
[Thread 0] group 20 finished
[Thread 0] group 21 finished
[Thread 0] group 22 finished
[Thread 0] group 23 finished
[Thread 0] group 24 finished
[Thread 0] group 25 finished
[Thread 0] group 26 finished
[Thread 0] group 27 finished
[Thread 0] group 28 finished
[Thread 0] group 29 finished
[Thread 0] group 30 finished
[Thread 0] group 31 finished
[Thread 0] group 32 finished
[Thread 0] Pass 1: Memory used: 472k/0k (345k/128k), time:  0.01/ 0.01/ 0.00
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 335.12MB/s
[Thread 0] Scanned group range [0, 32), inodes 402
Pass 2: Checking directory structure
Pass 2: Memory used: 684k/0k (299k/386k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 105.59MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 684k/0k (299k/386k), time:  0.05/ 0.02/ 0.01
Pass 3A: Memory used: 684k/0k (299k/386k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 684k/0k (296k/389k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 2127.66MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 568k/0k (67k/502k), time:  0.02/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 568k/0k (65k/504k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 225.63MB/s

         402 inodes used (0.06%, out of 720128)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 396
      130875 blocks used (12.48%, out of 1048576)
           0 bad blocks
           1 large file

         220 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         392 files
Memory used: 568k/0k (64k/505k), time:  0.08/ 0.04/ 0.01
I/O read: 2MB, write: 1MB, rate: 26.27MB/s
oleg422-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/mds1_flakey to 377837 (4k) blocks.
The filesystem on /dev/mapper/mds1_flakey is now 377837 (4k) blocks long.

oleg422-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/ost1_flakey to 591846 (4k) blocks.
The filesystem on /dev/mapper/ost1_flakey is now 589824 (4k) blocks long.

e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg422-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg422-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 20)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 24054 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24055 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24056 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24057 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24058 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24060 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48009 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] group 19 finished
[Thread 0] group 20 finished
[Thread 0] Pass 1: Memory used: 400k/0k (269k/132k), time:  0.00/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 215.01MB/s
[Thread 0] Scanned group range [0, 20), inodes 383
Pass 2: Checking directory structure
Pass 2: Memory used: 576k/0k (223k/354k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 147.71MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 576k/0k (224k/353k), time:  0.04/ 0.02/ 0.00
Pass 3A: Memory used: 576k/0k (224k/353k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 576k/0k (222k/355k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 2061.86MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 500k/0k (68k/433k), time:  0.02/ 0.01/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 500k/0k (67k/434k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 301.57MB/s

         382 inodes used (0.08%, out of 480000)
           4 non-contiguous files (1.0%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
      124660 blocks used (32.99%, out of 377837)
           0 bad blocks
           1 large file

         254 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         372 files
Memory used: 500k/0k (66k/435k), time:  0.06/ 0.04/ 0.00
I/O read: 1MB, write: 1MB, rate: 15.77MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg422-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg422-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 18)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] Pass 1: Memory used: 372k/0k (246k/127k), time:  0.01/ 0.00/ 0.01
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 368.92MB/s
[Thread 0] Scanned group range [0, 18), inodes 402
Pass 2: Checking directory structure
Pass 2: Memory used: 532k/0k (200k/333k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 101.10MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 532k/0k (200k/333k), time:  0.04/ 0.02/ 0.01
Pass 3A: Memory used: 532k/0k (200k/332k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 532k/0k (198k/335k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 1506.02MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 468k/0k (66k/403k), time:  0.02/ 0.01/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 468k/0k (65k/404k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 182.82MB/s

         402 inodes used (0.10%, out of 405072)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 396
       89417 blocks used (15.16%, out of 589824)
           0 bad blocks
           1 large file

         220 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         392 files
Memory used: 468k/0k (64k/405k), time:  0.07/ 0.03/ 0.01
I/O read: 2MB, write: 1MB, rate: 29.17MB/s
start mds service on oleg422-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=131072
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
check files after shrinking the MDT and OST filesystems
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-1 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-1 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-2 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-2 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-3 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-3 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-4 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-4 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-5 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-5 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-6 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-6 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-7 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-7 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-8 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-8 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-9 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-9 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-10 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-10 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-11 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-11 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-12 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-12 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-13 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-13 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-14 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-14 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-15 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-15 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-16 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-16 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-17 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-17 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-18 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-18 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-19 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-19 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-20 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-20 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-21 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-21 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-22 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-22 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-23 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-23 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-24 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-24 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-25 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-25 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-26 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-26 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-27 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-27 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-28 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-28 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-29 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-29 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-30 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-30 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-31 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-31 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-32 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-32 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-33 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-33 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-34 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-34 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-35 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-35 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-36 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-36 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-37 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-37 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-38 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-38 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-39 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-39 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-40 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-40 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-41 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-41 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-42 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-42 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-43 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-43 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-44 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-44 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-45 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-45 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-46 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-46 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-47 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-47 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-48 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-48 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-49 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-49 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-50 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-50 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-51 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-51 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-52 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-52 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-53 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-53 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-54 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-54 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-55 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-55 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-56 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-56 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-57 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-57 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-58 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-58 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-59 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-59 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-60 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-60 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-61 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-61 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-62 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-62 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-63 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-63 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-64 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-64 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-65 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-65 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-66 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-66 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-67 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-67 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-68 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-68 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-69 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-69 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-70 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-70 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-71 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-71 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-72 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-72 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-73 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-73 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-74 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-74 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-75 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-75 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-76 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-76 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-77 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-77 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-78 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-78 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-79 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-79 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-80 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-80 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-81 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-81 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-82 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-82 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-83 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-83 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-84 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-84 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-85 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-85 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-86 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-86 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-87 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-87 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-88 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-88 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-89 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-89 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-90 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-90 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-91 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-91 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-92 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-92 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-93 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-93 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-94 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-94 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-95 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-95 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-96 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-96 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-97 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-97 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-98 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-98 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-99 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-99 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-100 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-100 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-101 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-101 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-102 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-102 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-103 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-103 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-104 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-104 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-105 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-105 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-106 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-106 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-107 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-107 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-108 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-108 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-109 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-109 has size 1048576 OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-110 has type file OK
/mnt/lustre/d78.conf-sanity-1/f78.conf-sanity-1-110 has size 1048576 OK
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 78 (204s)

== conf-sanity-1 test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:33:31 (1734258811)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: 
oleg422-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg422-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22
oleg422-server: 
oleg422-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg422-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22
oleg422-server: 
oleg422-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg422-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 79 (60s)

== conf-sanity-1 test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 05:34:31 (1734258871)
mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem
mnt_opts -o loop
run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1
oleg422-server: oleg422-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no
oleg422-server: 
oleg422-server: llverfs: write /mnt/lustre-ost1/llverfs_dir00142/file000@0+1048576 short: 368640 written
oleg422-server: Timestamp: 1734258875
oleg422-server: dirs: 147, fs blocks: 37602
oleg422-server: write_done: /mnt/lustre-ost1/llverfs_dir00142/file000, current: 253.948 MB/s, overall: 253.948 MB/s, ETA: 0:00:00
oleg422-server: 
oleg422-server: read_done: /mnt/lustre-ost1/llverfs_dir00141/file000, current: 3074.33 MB/s, overall: 3074.33 MB/s, ETA: 0:00:00
oleg422-server: 
unmount the OST /dev/mapper/ost1_flakey
Stopping /mnt/lustre-ost1 (opts:) on oleg422-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg422-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: No space left on device
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 28
oleg422-server: error: set_param: param_path 'seq/cli-lustre': No such file or directory
oleg422-server: error: set_param: setting 'seq/cli-lustre'='OST0000-super.width=65536': No such file or directory
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 28
string err 
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 83 (61s)

== conf-sanity-1 test 86: Replacing mkfs.lustre -G option ========================================================== 05:35:33 (1734258933)
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps
params: --mgsnode=192.168.204.122@tcp --fsname=lustre --ost 			--index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-G 1024 -b 4096 -O flex_bg
 -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey 
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST0000
	kilobytes     200000
	options       -G 1024 -b 4096   -I 512 -q -O flex_bg,uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -G 1024 -b 4096   -I 512 -q -O flex_bg,uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 86 (53s)

== conf-sanity-1 test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 05:36:26 (1734258986)
Estimate: at most 351-byte space left in inode.
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
setup single mount lustre success
4 -rw-r--r-- 1 root root 67108865 Dec 15 05:37 /mnt/lustre-mds1/ROOT/f87.conf-sanity-1
Verified: at most 351-byte space left in inode.
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
PASS 87 (55s)

== conf-sanity-1 test 89: check tunefs --param and --erase-param{s} options ========================================================== 05:37:21 (1734259041)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
tunefs --param failover.node=192.0.2.254@tcp0
tunefs --param failover.node=192.0.2.255@tcp0
tunefs --erase-param failover.node
tunefs --erase-params
tunefs --param failover.node=192.0.2.254@tcp0 --erase-params
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 89 (56s)

== conf-sanity-1 test 93: register mulitple MDT at the same time ========================================================== 05:38:17 (1734259097)

 SKIP: conf-sanity-1 test_93 needs >= 3 MDTs
SKIP 93 (1s)

== conf-sanity-1 test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:38:18 (1734259098)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
setup single mount lustre success
lustre-MDT0000:
lustre-OST0000:
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 100 (49s)

== conf-sanity-1 test 104a: Make sure user defined options are reflected in mount ========================================================== 05:39:07 (1734259147)
mountfsopt: acl,user_xattr
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
Starting mds1: -o localrecov,noacl  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
Starting mds2: -o localrecov,noacl  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
setfacl: /mnt/lustre: Operation not supported
PASS 104a (45s)

== conf-sanity-1 test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:39:52 (1734259192)
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
oleg422-server: rmmod: ERROR: Module lustre is in use
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
touch: cannot touch '/tmp/d105.conf-sanity-1/f105.conf-sanity-1': Read-only file system
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 105 (95s)

== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:41:27 (1734259287)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992         274       79718   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896         368       32528   2% /mnt/lustre[OST:0]

filesystem_summary:        33057         529       32528   2% /mnt/lustre

seq.cli-lustre-OST0000-super.width=0x1ffffff
fail_loc=0x1312
fail_val=5
 - open/close 5173 (time 1734259338.30 total 10.00 last 517.28)
 - open/close 10000 (time 1734259348.08 total 19.78 last 493.52)
 - open/close 14919 (time 1734259358.08 total 29.78 last 491.87)
 - open/close 20000 (time 1734259366.53 total 38.23 last 601.71)
 - open/close 25300 (time 1734259376.53 total 48.23 last 529.94)
 - open/close 29579 (time 1734259386.53 total 58.23 last 427.83)
open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device
total: 32560 open/close in 64.86 seconds: 502.02 ops/second
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992       32835       47157  42% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896       32896           0 100% /mnt/lustre[OST:0]

filesystem_summary:        33090       33090           0 100% /mnt/lustre

 - /unlink 8946 (time 1734259404.09 total 10.00 last 894.57)
 - /unlink 18201 (time 1734259414.09 total 20.00 last 925.40)
 - /unlink 20000 (time 1734259416.37 total 22.29 last 787.62)
 - /unlink 29729 (time 1734259426.37 total 32.29 last 972.84)
unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory
total: 32560 /unlink in 35.07 seconds: 928.56 ops/second
Waiting for MDT destroys to complete
oleg422-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg422-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
 - open/close 5025 (time 1734259466.01 total 10.00 last 502.49)
 - open/close 10000 (time 1734259475.95 total 19.94 last 500.36)
 - open/close 15241 (time 1734259485.95 total 29.94 last 524.08)
 - open/close 20000 (time 1734259493.68 total 37.67 last 615.60)
 - open/close 26097 (time 1734259503.68 total 47.68 last 609.61)
 - open/close 30000 (time 1734259510.14 total 54.13 last 604.36)
open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device
total: 32560 open/close in 58.51 seconds: 556.46 ops/second
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992       32835       47157  42% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896       32896           0 100% /mnt/lustre[OST:0]

filesystem_summary:        33090       33090           0 100% /mnt/lustre

 - /unlink 10000 (time 1734259524.64 total 9.32 last 1073.21)
 - /unlink 20000 (time 1734259534.08 total 18.76 last 1059.23)
 - /unlink 30000 (time 1734259542.90 total 27.58 last 1133.97)
unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory
total: 32560 /unlink in 29.80 seconds: 1092.51 ops/second
Waiting for MDT destroys to complete
oleg422-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg422-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
 - open/close 5843 (time 1734259585.56 total 10.00 last 584.27)
 - open/close 10000 (time 1734259592.78 total 17.22 last 575.52)
 - open/close 16134 (time 1734259602.79 total 27.22 last 613.34)
 - open/close 20000 (time 1734259608.86 total 33.30 last 636.19)
 - open/close 26123 (time 1734259618.86 total 43.30 last 612.21)
 - open/close 30000 (time 1734259625.03 total 49.47 last 628.68)
open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device
total: 32560 open/close in 53.55 seconds: 608.07 ops/second
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992       32835       47157  42% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896       32896           0 100% /mnt/lustre[OST:0]

filesystem_summary:        33090       33090           0 100% /mnt/lustre

 - /unlink 10000 (time 1734259639.08 total 9.14 last 1093.56)
 - /unlink 20000 (time 1734259648.74 total 18.81 last 1035.11)
 - /unlink 30000 (time 1734259658.60 total 28.66 last 1014.23)
unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory
total: 32560 /unlink in 31.43 seconds: 1036.04 ops/second
Waiting for MDT destroys to complete
oleg422-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg422-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
 - open/close 6416 (time 1734259722.07 total 10.00 last 641.60)
 - open/close 10000 (time 1734259728.19 total 16.12 last 585.36)
 - open/close 15334 (time 1734259738.19 total 26.12 last 533.35)
 - open/close 20000 (time 1734259745.60 total 33.53 last 629.85)
 - open/close 26111 (time 1734259755.60 total 43.53 last 611.03)
 - open/close 30000 (time 1734259762.78 total 50.71 last 541.80)
open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device
total: 32560 open/close in 55.98 seconds: 581.59 ops/second
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992       32835       47157  42% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896       32896           0 100% /mnt/lustre[OST:0]

filesystem_summary:        33090       33090           0 100% /mnt/lustre

 - /unlink 9058 (time 1734259778.97 total 10.00 last 905.77)
 - /unlink 18212 (time 1734259788.97 total 20.00 last 915.31)
 - /unlink 28259 (time 1734259798.97 total 30.00 last 1004.68)
unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory
total: 32560 /unlink in 34.14 seconds: 953.80 ops/second
Waiting for MDT destroys to complete
oleg422-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg422-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
 - open/close 5353 (time 1734259837.37 total 10.00 last 535.22)
 - open/close 10000 (time 1734259846.49 total 19.11 last 509.94)
 - open/close 15690 (time 1734259856.49 total 29.12 last 568.95)
 - open/close 20000 (time 1734259864.16 total 36.78 last 562.03)
 - open/close 25417 (time 1734259874.16 total 46.79 last 541.63)
 - open/close 30000 (time 1734259882.40 total 55.03 last 555.82)
open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device
total: 32560 open/close in 59.70 seconds: 545.36 ops/second
UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        79992       32760       47232  41% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        79992         255       79737   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID        32896       32896           0 100% /mnt/lustre[OST:0]

filesystem_summary:        33015       33015           0 100% /mnt/lustre

 - /unlink 9712 (time 1734259897.97 total 10.00 last 971.13)
 - /unlink 19771 (time 1734259907.97 total 20.00 last 1005.85)
 - /unlink 29806 (time 1734259917.97 total 30.00 last 1003.50)
unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory
total: 32560 /unlink in 32.69 seconds: 996.08 ops/second
Waiting for MDT destroys to complete
oleg422-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg422-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
fail_loc=0
fail_val=0
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 106 (680s)

== conf-sanity-1 test 109a: test lctl clear_conf fsname == 05:52:47 (1734259967)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Setting lustre-MDT0000.mdd.atime_diff from 60 to 62
Waiting 90s for '62'
Updated after 4s: want '62' got '62'
Setting lustre-MDT0000.mdd.atime_diff from 62 to 63
Waiting 90s for '63'
Updated after 5s: want '63' got '63'
Setting lustre.llite.max_read_ahead_mb from 256 to 32
Waiting 90s for '32'
Setting lustre.llite.max_read_ahead_mb from 32 to 64
Waiting 90s for '64'
Updated after 6s: want '64' got '64'
oleg422-server: Pool lustre.pool1 created
Waiting 90s for ''
oleg422-server: OST lustre-OST0000_UUID added to pool lustre.pool1
oleg422-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg422-server: OST lustre-OST0000_UUID added to pool lustre.pool1
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
start mds service on oleg422-server
Starting mds1: -o localrecov -o nosvc  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
Start /dev/mapper/mds1_flakey without service
Started lustre-MDT0000
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Destroy the created pools: pool1
lustre.pool1
oleg422-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg422-server: Pool lustre.pool1 destroyed
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 109a (164s)

== conf-sanity-1 test 109b: test lctl clear_conf one config ========================================================== 05:55:31 (1734260131)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Setting lustre-MDT0000.mdd.atime_diff from 60 to 62
Setting lustre-MDT0000.mdd.atime_diff from 62 to 63
Waiting 90s for '63'
Updated after 9s: want '63' got '63'
Setting lustre.llite.max_read_ahead_mb from 256 to 32
Waiting 90s for '32'
Updated after 8s: want '32' got '32'
Setting lustre.llite.max_read_ahead_mb from 32 to 64
Waiting 90s for '64'
Updated after 7s: want '64' got '64'
oleg422-server: Pool lustre.pool1 created
Waiting 90s for ''
oleg422-server: OST lustre-OST0000_UUID added to pool lustre.pool1
oleg422-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg422-server: OST lustre-OST0000_UUID added to pool lustre.pool1
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
start mds service on oleg422-server
Starting mds1: -o localrecov -o nosvc  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
Start /dev/mapper/mds1_flakey without service
Started lustre-MDT0000
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Destroy the created pools: pool1
lustre.pool1
oleg422-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg422-server: Pool lustre.pool1 destroyed
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
PASS 109b (166s)

== conf-sanity-1 test 116: big size MDT support ========== 05:58:17 (1734260297)
/usr/sbin/mkfs.xfs
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
meta-data=/tmp/f116.conf-sanity-1-mdt0 isize=512    agcount=4, agsize=67108864 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=268435456, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=131072, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.204.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: not found
formatting backing filesystem ldiskfs on /dev/loop1
	target name   lustre:MDT0000
	kilobytes     18253611008
	options       -i 16777216 -b 4096  -J size=4096 -I 1024 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,64bit,^resize_inode,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,lazy_itable_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -i 16777216 -b 4096  -J size=4096 -I 1024 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,64bit,^resize_inode,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,lazy_itable_init,packed_meta_blocks -F /dev/loop1 18253611008k
Writing CONFIGS/mountdata
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 116 (70s)

== conf-sanity-1 test 122a: Check OST sequence update ==== 05:59:27 (1734260367)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
fail_loc=0x00001e0
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
fail_loc=0
total: 1000 open/close in 2.80 seconds: 357.17 ops/second
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 122a (75s)

== conf-sanity-1 test 123ah: del_ost cancels config log entries correctly ========================================================== 06:00:42 (1734260442)
Checking servers environments
Checking clients oleg422-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Starting client oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Started clients oleg422-client.virtnet: 
192.168.204.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800ab8dd800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800ab8dd800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 2s: want 'procname_uid' got 'procname_uid'
disable quota as required
del_ost: dry run for target lustre-OST0000
config_log: lustre-MDT0000
[DRY RUN] cancel catalog 'lustre-MDT0000:29':"- { index: 29, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-MDT0000:28':"- { index: 28, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.204.122@tcp }"
[DRY RUN] cancel catalog 'lustre-MDT0000:27':"- { index: 27, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-client
[DRY RUN] cancel catalog 'lustre-client:34':"- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-client:33':"- { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.204.122@tcp }"
[DRY RUN] cancel catalog 'lustre-client:32':"- { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-MDT0001
[DRY RUN] cancel catalog 'lustre-MDT0001:32':"- { index: 32, event: add_osc, device: lustre-MDT0001-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-MDT0001:31':"- { index: 31, event: setup, device: lustre-OST0000-osc-MDT0001, UUID: lustre-OST0000_UUID, node: 192.168.204.122@tcp }"
[DRY RUN] cancel catalog 'lustre-MDT0001:30':"- { index: 30, event: attach, device: lustre-OST0000-osc-MDT0001, type: osc, UUID: lustre-MDT0001-mdtlov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-MDT0000
cancel catalog lustre-MDT0000 log_idx 29: done
cancel catalog lustre-MDT0000 log_idx 28: done
cancel catalog lustre-MDT0000 log_idx 27: done
del_ost: cancelled 3 catalog entries
config_log: lustre-client
cancel catalog lustre-client log_idx 34: done
cancel catalog lustre-client log_idx 33: done
cancel catalog lustre-client log_idx 32: done
del_ost: cancelled 3 catalog entries
config_log: lustre-MDT0001
cancel catalog lustre-MDT0001 log_idx 32: done
cancel catalog lustre-MDT0001 log_idx 31: done
cancel catalog lustre-MDT0001 log_idx 30: done
del_ost: cancelled 3 catalog entries
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
oleg422-server: rmmod: ERROR: Module lustre is in use
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
fail_loc=0
PASS 123ah (120s)

== conf-sanity-1 test 132: hsm_actions processed after failover ========================================================== 06:02:42 (1734260562)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mdt.hsm_control=enabled

Writing CONFIGS/mountdata
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
PASS 132 (27s)

== conf-sanity-1 test 133: stripe QOS: free space balance in a pool ========================================================== 06:03:09 (1734260589)

 SKIP: conf-sanity-1 test_133 needs >= 4 OSTs
SKIP 133 (0s)

== conf-sanity-1 test 135: check the behavior when changelog is wrapped around ========================================================== 06:03:09 (1734260589)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
oleg422-client: fail_loc=0x1312
oleg422-client: fail_val=5
oleg422-server: fail_loc=0x1312
oleg422-server: fail_val=5
striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity-1
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 4211 (time 1734260636.36 total 10.00 last 421.08)
total: 4500 open/close in 10.91 seconds: 412.35 ops/second
total: 4500 /unlink in 7.08 seconds: 635.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 4290 (time 1734260656.85 total 10.00 last 428.96)
total: 4500 open/close in 10.56 seconds: 426.32 ops/second
total: 4500 /unlink in 6.43 seconds: 699.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 3529 (time 1734260675.79 total 10.00 last 352.89)
total: 4500 open/close in 12.18 seconds: 369.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
total: 4500 /unlink in 7.34 seconds: 612.84 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 10973: 23915 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg422-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
PASS 135 (132s)

== conf-sanity-1 test 136: don't panic with bad obdecho setup ========================================================== 06:05:21 (1734260721)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
oleg422-server: oleg422-server.virtnet: executing load_module obdecho/obdecho
oleg422-server: error: setup: Invalid argument
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22
oleg422-server: error: test_mkdir: No such file or directory
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping client oleg422-client.virtnet /mnt/lustre opts:
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 136 (117s)

== conf-sanity-1 test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:07:18 (1734260838)

 SKIP: conf-sanity-1 test_137 needs >= 3 MDTs
SKIP 137 (1s)

== conf-sanity-1 test 140: remove_updatelog script actions ========================================================== 06:07:19 (1734260839)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
striped dir -i0 -c2 -H crush2 /mnt/lustre/d140.conf-sanity-1
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Dry run was requested, no changes will be applied
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 0 1
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
Processing MDT1 llog catalog [0x240000400:0x1:0x0] ...
remove_updatelog: /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] is too small.
> /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0]
Dry run was requested, no changes will be applied
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 1 0
Processing MDT1 llog catalog [0x240000400:0x1:0x0] ...
remove_updatelog: /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] is too small.
> /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0]
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 0
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping client oleg422-client.virtnet /mnt/lustre opts:
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 140 (224s)

== conf-sanity-1 test 151a: damaged local config doesn't prevent mounting ========================================================== 06:11:03 (1734261063)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg422-server
stop mds service on oleg422-server
stop mds service on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
Damage ost1 local config log
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
start ost1 service on oleg422-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg422-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Invalid argument
oleg422-server: This may have multiple causes.
oleg422-server: Are the mount options correct?
oleg422-server: Check the syslog for more info.
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22
oleg422-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg422-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 22
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 151a (137s)

== conf-sanity-1 test 151b: -ENOSPC doesn't affect mount ========================================================== 06:13:20 (1734261200)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg422-server
stop mds service on oleg422-server
stop mds service on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
start mds service on oleg422-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
fail_loc=0x131e
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg422-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
PASS 151b (105s)

== conf-sanity-1 test 153a: bypass invalid NIDs quickly == 06:15:05 (1734261305)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
waiting for mount ...
waiting for mount ...
waiting for mount ...
          "192.168.204.122@tcp": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 10 }
          "192.168.252.112@tcp": { connects: 1, replied: 0, uptodate: discovering, sec_ago: 15 }
          "10.252.252.113@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
          "192.168.204.122@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
setup single mount lustre success
umount lustre on /mnt/lustre.....
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 153a (200s)

== conf-sanity-1 test 153b: added IPv6 NID support ======= 06:18:25 (1734261505)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Checking servers environments
Checking clients oleg422-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Starting client oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Started clients oleg422-client.virtnet: 
192.168.204.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800aa24b000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800aa24b000.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 4s: want 'procname_uid' got 'procname_uid'
disable quota as required
PASS 153b (113s)

== conf-sanity-1 test 153c: don't stuck on unreached NID ========================================================== 06:20:18 (1734261618)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg422-server: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg422-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
start mds service on oleg422-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg422-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg422-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
          "192.168.252.112@tcp": { connects: 3, replied: 0, uptodate: discovering, sec_ago: 0 }
          "192.168.252.113@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 10 }
          "192.168.252.115@tcp": { connects: 2, replied: 0, uptodate: discovering, sec_ago: 5 }
connection attempts: 7
Waiting for mount to fail
mount.lustre: mount 192.168.252.112@tcp:192.168.252.113@tcp:192.168.252.115@tcp:/lustre at /mnt/lustre failed: Input/output error
Is the MGS running?
umount lustre on /mnt/lustre.....
stop ost1 service on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
stop mds service on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
unloading modules via unload_modules_local on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
PASS 153c (366s)

== conf-sanity-1 test 154: expand .. on rename after MDT backup restore ========================================================== 06:26:24 (1734261984)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
Checking servers environments
Checking clients oleg422-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Starting client oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Started clients oleg422-client.virtnet: 
192.168.204.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b378b800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b378b800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 4s: want 'procname_uid' got 'procname_uid'
disable quota as required
striped dir -i0 -c1 -H crush2 /mnt/lustre/d154.conf-sanity-1
striped dir -i0 -c1 -H all_char /mnt/lustre/d154.conf-sanity-1.tgt
striped dir -i0 -c1 -H crush2 /mnt/lustre/d154.conf-sanity-1.dx_dir
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d154.conf-sanity-1.dirent
striped dir -i0 -c1 -H all_char /mnt/lustre/d154.conf-sanity-1.empty
total: 128 open/close in 0.53 seconds: 242.27 ops/second
total: 2 open/close in 0.03 seconds: 79.20 ops/second
==target parent FID: fid:[0x200000402:0x1:0x0]==
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping client oleg422-client.virtnet /mnt/lustre opts:
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
==debugfs before backup==
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 53385   Type: directory    Mode:  0755   Flags: 0x1000
Generation: 413538329    Version: 0x00000001:0000010c
User:     0   Group:     0   Project:     0   Size: 12288
File ACL: 0
Links: 2   Blockcount: 24
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebd25:00000000 -- Sun Dec 15 06:27:33 2024
 atime: 0x675ebd23:00000000 -- Sun Dec 15 06:27:31 2024
 mtime: 0x675ebd25:00000000 -- Sun Dec 15 06:27:33 2024
crtime: 0x675ebd24:53354500 -- Sun Dec 15 06:27:32 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x3:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  user.job (5) = "lfs.0"
  linkea: idx=0 parent=[0x200000402:0x1:0x0] name='d154.conf-sanity-1.dx_dir'
BLOCKS:
(0):24781, (1-2):24784-24785
TOTAL: 3

oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  53385   40755 (2)      0      0   12288 15-Dec-2024 06:27 .
  53383   40755 (18)      0      0    4096 15-Dec-2024 06:27 fid:[0x200000402:0x1:0x0] ..
    168  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x9:0x0] f154.conf-sanity-1.longfilename3
    169  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0xa:0x0] f154.conf-sanity-1.longfilename4
    170  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0xb:0x0] f154.conf-sanity-1.longfilename5
    171  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0xc:0x0] f154.conf-sanity-1.longfilename6
    172  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0xd:0x0] f154.conf-sanity-1.longfilename7
    175  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x10:0x0] f154.conf-sanity-1.longfilename10
    177  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x12:0x0] f154.conf-sanity-1.longfilename12
    178  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x13:0x0] f154.conf-sanity-1.longfilename13
    179  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x14:0x0] f154.conf-sanity-1.longfilename14
    182  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x17:0x0] f154.conf-sanity-1.longfilename17
    184  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x19:0x0] f154.conf-sanity-1.longfilename19
    188  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x1d:0x0] f154.conf-sanity-1.longfilename23
    189  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x1e:0x0] f154.conf-sanity-1.longfilename24
    190  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x1f:0x0] f154.conf-sanity-1.longfilename25
    193  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x22:0x0] f154.conf-sanity-1.longfilename28
    195  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x24:0x0] f154.conf-sanity-1.longfilename30
    196  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x25:0x0] f154.conf-sanity-1.longfilename31
    198  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x27:0x0] f154.conf-sanity-1.longfilename33
    199  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x28:0x0] f154.conf-sanity-1.longfilename34
    201  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x2a:0x0] f154.conf-sanity-1.longfilename36
    202  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x2b:0x0] f154.conf-sanity-1.longfilename37
    203  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x2c:0x0] f154.conf-sanity-1.longfilename38
    204  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x2d:0x0] f154.conf-sanity-1.longfilename39
    206  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x2f:0x0] f154.conf-sanity-1.longfilename41
    209  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x32:0x0] f154.conf-sanity-1.longfilename44
    210  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x33:0x0] f154.conf-sanity-1.longfilename45
    214  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x37:0x0] f154.conf-sanity-1.longfilename49
    215  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x38:0x0] f154.conf-sanity-1.longfilename50
    217  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x3a:0x0] f154.conf-sanity-1.longfilename52
    221  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x3e:0x0] f154.conf-sanity-1.longfilename56
    224  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x41:0x0] f154.conf-sanity-1.longfilename59
    229  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x46:0x0] f154.conf-sanity-1.longfilename64
    231  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x48:0x0] f154.conf-sanity-1.longfilename66
    232  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x49:0x0] f154.conf-sanity-1.longfilename67
    233  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x4a:0x0] f154.conf-sanity-1.longfilename68
    237  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x4e:0x0] f154.conf-sanity-1.longfilename72
    238  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x4f:0x0] f154.conf-sanity-1.longfilename73
    240  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x51:0x0] f154.conf-sanity-1.longfilename75
    241  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x52:0x0] f154.conf-sanity-1.longfilename76
    243  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x54:0x0] f154.conf-sanity-1.longfilename78
    244  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x55:0x0] f154.conf-sanity-1.longfilename79
    247  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x58:0x0] f154.conf-sanity-1.longfilename82
    249  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x5a:0x0] f154.conf-sanity-1.longfilename84
    251  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x5c:0x0] f154.conf-sanity-1.longfilename86
    252  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x5d:0x0] f154.conf-sanity-1.longfilename87
    254  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x5f:0x0] f154.conf-sanity-1.longfilename89
    257  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x62:0x0] f154.conf-sanity-1.longfilename92
    260  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x65:0x0] f154.conf-sanity-1.longfilename95
    262  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x67:0x0] f154.conf-sanity-1.longfilename97
    263  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x68:0x0] f154.conf-sanity-1.longfilename98
    267  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x6c:0x0] f154.conf-sanity-1.longfilename102
    270  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x6f:0x0] f154.conf-sanity-1.longfilename105
    271  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x70:0x0] f154.conf-sanity-1.longfilename106
    272  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x71:0x0] f154.conf-sanity-1.longfilename107
    273  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x72:0x0] f154.conf-sanity-1.longfilename108
    274  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x73:0x0] f154.conf-sanity-1.longfilename109
    275  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x74:0x0] f154.conf-sanity-1.longfilename110
    277  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x76:0x0] f154.conf-sanity-1.longfilename112
    278  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x77:0x0] f154.conf-sanity-1.longfilename113
    282  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x7b:0x0] f154.conf-sanity-1.longfilename117
    283  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x7c:0x0] f154.conf-sanity-1.longfilename118
    284  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x7d:0x0] f154.conf-sanity-1.longfilename119
    285  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x7e:0x0] f154.conf-sanity-1.longfilename120
    286  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x7f:0x0] f154.conf-sanity-1.longfilename121
    289  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x82:0x0] f154.conf-sanity-1.longfilename124
    290  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x83:0x0] f154.conf-sanity-1.longfilename125
    291  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x84:0x0] f154.conf-sanity-1.longfilename126
    292  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x85:0x0] f154.conf-sanity-1.longfilename127
    225  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x42:0x0] f154.conf-sanity-1.longfilename60
    222  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x3f:0x0] f154.conf-sanity-1.longfilename57
    208  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x31:0x0] f154.conf-sanity-1.longfilename43
    181  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x16:0x0] f154.conf-sanity-1.longfilename16
    219  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x3c:0x0] f154.conf-sanity-1.longfilename54
    192  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x21:0x0] f154.conf-sanity-1.longfilename27
    212  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x35:0x0] f154.conf-sanity-1.longfilename47
    166  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x7:0x0] f154.conf-sanity-1.longfilename1
    226  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x43:0x0] f154.conf-sanity-1.longfilename61
    213  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x36:0x0] f154.conf-sanity-1.longfilename48
    218  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x3b:0x0] f154.conf-sanity-1.longfilename53
    191  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x20:0x0] f154.conf-sanity-1.longfilename26
    216  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x39:0x0] f154.conf-sanity-1.longfilename51
    207  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x30:0x0] f154.conf-sanity-1.longfilename42
    211  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x34:0x0] f154.conf-sanity-1.longfilename46
    187  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x1c:0x0] f154.conf-sanity-1.longfilename22
    167  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x8:0x0] f154.conf-sanity-1.longfilename2
    197  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x26:0x0] f154.conf-sanity-1.longfilename32
    200  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x29:0x0] f154.conf-sanity-1.longfilename35
    174  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0xf:0x0] f154.conf-sanity-1.longfilename9
    228  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x45:0x0] f154.conf-sanity-1.longfilename63
    227  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x44:0x0] f154.conf-sanity-1.longfilename62
    186  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x1b:0x0] f154.conf-sanity-1.longfilename21
    194  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x23:0x0] f154.conf-sanity-1.longfilename29
    220  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x3d:0x0] f154.conf-sanity-1.longfilename55
    165  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x6:0x0] f154.conf-sanity-1.longfilename0
    183  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x18:0x0] f154.conf-sanity-1.longfilename18
    205  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x2e:0x0] f154.conf-sanity-1.longfilename40
    173  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0xe:0x0] f154.conf-sanity-1.longfilename8
    230  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x47:0x0] f154.conf-sanity-1.longfilename65
    176  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x11:0x0] f154.conf-sanity-1.longfilename11
    180  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x15:0x0] f154.conf-sanity-1.longfilename15
    185  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x1a:0x0] f154.conf-sanity-1.longfilename20
    223  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x40:0x0] f154.conf-sanity-1.longfilename58
    234  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x4b:0x0] f154.conf-sanity-1.longfilename69
    235  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x4c:0x0] f154.conf-sanity-1.longfilename70
    236  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x4d:0x0] f154.conf-sanity-1.longfilename71
    239  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x50:0x0] f154.conf-sanity-1.longfilename74
    242  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x53:0x0] f154.conf-sanity-1.longfilename77
    245  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x56:0x0] f154.conf-sanity-1.longfilename80
    246  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x57:0x0] f154.conf-sanity-1.longfilename81
    248  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x59:0x0] f154.conf-sanity-1.longfilename83
    250  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x5b:0x0] f154.conf-sanity-1.longfilename85
    253  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x5e:0x0] f154.conf-sanity-1.longfilename88
    255  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x60:0x0] f154.conf-sanity-1.longfilename90
    256  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x61:0x0] f154.conf-sanity-1.longfilename91
    258  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x63:0x0] f154.conf-sanity-1.longfilename93
    259  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x64:0x0] f154.conf-sanity-1.longfilename94
    261  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x66:0x0] f154.conf-sanity-1.longfilename96
    264  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x69:0x0] f154.conf-sanity-1.longfilename99
    265  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x6a:0x0] f154.conf-sanity-1.longfilename100
    266  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x6b:0x0] f154.conf-sanity-1.longfilename101
    268  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x6d:0x0] f154.conf-sanity-1.longfilename103
    269  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x6e:0x0] f154.conf-sanity-1.longfilename104
    276  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x75:0x0] f154.conf-sanity-1.longfilename111
    279  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x78:0x0] f154.conf-sanity-1.longfilename114
    280  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x79:0x0] f154.conf-sanity-1.longfilename115
    281  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x7a:0x0] f154.conf-sanity-1.longfilename116
    287  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x80:0x0] f154.conf-sanity-1.longfilename122
    288  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x81:0x0] f154.conf-sanity-1.longfilename123

oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 53386   Type: directory    Mode:  0755   Flags: 0x0
Generation: 413538330    Version: 0x00000001:0000010d
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebd25:00000000 -- Sun Dec 15 06:27:33 2024
 atime: 0x675ebd23:00000000 -- Sun Dec 15 06:27:31 2024
 mtime: 0x675ebd25:00000000 -- Sun Dec 15 06:27:33 2024
crtime: 0x675ebd24:60152b00 -- Sun Dec 15 06:27:32 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x4:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  user.job (5) = "lfs.0"
  linkea: idx=0 parent=[0x200000402:0x1:0x0] name='d154.conf-sanity-1.dirent'
BLOCKS:
(0):24782
TOTAL: 1

oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  53386   40755 (2)      0      0    4096 15-Dec-2024 06:27 .
  53383   40755 (18)      0      0    4096 15-Dec-2024 06:27 fid:[0x200000402:0x1:0x0] ..
    293  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x86:0x0] f154.conf-sanity-1.longfilename0
    294  100644 (17)      0      0       0 15-Dec-2024 06:27 fid:[0x200000402:0x87:0x0] f154.conf-sanity-1.longfilename1

oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 53387   Type: directory    Mode:  0755   Flags: 0x0
Generation: 413538331    Version: 0x00000001:0000010e
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebd25:00000000 -- Sun Dec 15 06:27:33 2024
 atime: 0x675ebd23:00000000 -- Sun Dec 15 06:27:31 2024
 mtime: 0x675ebd23:00000000 -- Sun Dec 15 06:27:31 2024
crtime: 0x675ebd24:6cb80800 -- Sun Dec 15 06:27:32 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x5:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  user.job (5) = "lfs.0"
  linkea: idx=0 parent=[0x200000402:0x1:0x0] name='d154.conf-sanity-1.empty'
BLOCKS:
(0):24783
TOTAL: 1

oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  53387   40755 (2)      0      0    4096 15-Dec-2024 06:27 .
  53383   40755 (18)      0      0    4096 15-Dec-2024 06:27 fid:[0x200000402:0x1:0x0] ..

file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
Checking servers environments
Checking clients oleg422-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all
pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Starting client oleg422-client.virtnet:  -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre
Started clients oleg422-client.virtnet: 
192.168.204.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff88012a53b000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff88012a53b000.idle_timeout=debug
disable quota as required
==pre-rename parent/.. inodes==
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:27 /mnt/lustre/d154.conf-sanity-1
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:27 /mnt/lustre/d154.conf-sanity-1/d154.conf-sanity-1.dx_dir/..
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:27 /mnt/lustre/d154.conf-sanity-1
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:27 /mnt/lustre/d154.conf-sanity-1/d154.conf-sanity-1.dirent/..
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:27 /mnt/lustre/d154.conf-sanity-1
144115205289279489 drwxr-xr-x 5 root root 4096 Dec 15 06:27 /mnt/lustre/d154.conf-sanity-1/d154.conf-sanity-1.empty/..
==post-rename parent/.. inodes==
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:28 /mnt/lustre/d154.conf-sanity-1.tgt
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:28 /mnt/lustre/d154.conf-sanity-1.tgt/d154.conf-sanity-1.dx_dir/..
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:28 /mnt/lustre/d154.conf-sanity-1.tgt
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:28 /mnt/lustre/d154.conf-sanity-1.tgt/d154.conf-sanity-1.dirent/..
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:28 /mnt/lustre/d154.conf-sanity-1.tgt
144115205289279490 drwxr-xr-x 5 root root 4096 Dec 15 06:28 /mnt/lustre/d154.conf-sanity-1.tgt/d154.conf-sanity-1.empty/..
==target parent FID: fid:[0x200000402:0x2:0x0]==
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:)
Stopping client oleg422-client.virtnet /mnt/lustre opts:
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server
==post-rename .dx_dir should have 'fid:[0x200000402:0x2:0x0] ..'==
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 26740   Type: directory    Mode:  0755   Flags: 0x1000
Generation: 3278234573    Version: 0x00000002:00000001
User:     0   Group:     0   Project:     0   Size: 12288
File ACL: 0
Links: 2   Blockcount: 24
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebd72:00000000 -- Sun Dec 15 06:28:50 2024
 atime: 0x675ebd4e:29020c00 -- Sun Dec 15 06:28:14 2024
 mtime: 0x675ebd25:00000000 -- Sun Dec 15 06:27:33 2024
crtime: 0x675ebd4e:29020c00 -- Sun Dec 15 06:28:14 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x3:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  linkea: idx=0 parent=[0x200000402:0x2:0x0] name='d154.conf-sanity-1.dx_dir'
BLOCKS:
(0-2):24262-24264
TOTAL: 3

oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  26740   40755 (2)      0      0   12288 15-Dec-2024 06:27 .
  26738   40755 (18)      0      0    4096 15-Dec-2024 06:28 fid:[0x200000402:0x2:0x0] ..
    138  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename24
    140  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename6
    141  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename34
    144  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename12
    145  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename66
    146  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename49
    147  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename107
    148  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename92
    149  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename102
    150  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename84
    151  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename10
    152  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename64
    153  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename97
    155  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename52
    156  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename87
    157  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename36
    158  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename4
    160  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename120
    166  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename25
    167  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename78
    168  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename31
    172  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename28
    173  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename17
    176  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename124
    178  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename118
    180  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename121
    183  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename75
    184  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename33
    185  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename105
    186  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename73
    191  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename89
    194  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename7
    195  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename56
    198  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename19
    200  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename113
    201  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename98
    202  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename39
    205  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename71
    206  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename57
    207  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename43
    214  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename101
    215  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename103
    217  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename47
    218  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename1
    221  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename69
    225  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename51
    227  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename46
    229  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename83
    236  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename9
    238  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename85
    239  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename116
    240  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename99
    241  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename62
    243  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename115
    244  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename21
    245  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename74
    246  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename29
    252  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename0
    253  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename80
    254  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename18
    257  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename111
    258  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename65
    262  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename20
    263  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename58
    203  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename45
    181  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename95
    136  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename86
    213  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename93
    171  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename82
    187  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename37
    209  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename81
    143  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename76
    199  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename112
    162  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename14
    204  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename60
    197  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename59
    188  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename72
    228  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename22
    169  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename127
    179  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename38
    190  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename41
    175  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename109
    174  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename108
    226  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename42
    196  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename68
    193  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename126
    177  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename44
    216  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename122
    159  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename106
    142  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename3
    192  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename23
    137  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename79
    163  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename5
    182  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename110
    154  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename117
    222  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename53
    165  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename67
    210  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename54
    212  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename27
    164  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename30
    211  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename114
    189  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename125
    139  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename50
    170  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename119
    223  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename26
    220  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename48
    161  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename13
    224  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename123
    219  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename61
    208  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename16
    230  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename96
    231  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename2
    232  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename32
    233  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename91
    234  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename70
    235  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename35
    237  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename63
    242  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename104
    247  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename100
    248  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename94
    249  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename55
    250  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename90
    251  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename88
    255  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename40
    256  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename8
    259  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename11
    260  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename77
    261  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename15

==post-rename .dirent should have 'fid:[0x200000402:0x2:0x0] ..'==
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 26742   Type: directory    Mode:  0755   Flags: 0x0
Generation: 3278234703    Version: 0x00000002:00000002
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebd72:00000000 -- Sun Dec 15 06:28:50 2024
 atime: 0x675ebd4e:2d0fa500 -- Sun Dec 15 06:28:14 2024
 mtime: 0x675ebd25:00000000 -- Sun Dec 15 06:27:33 2024
crtime: 0x675ebd4e:2d0fa500 -- Sun Dec 15 06:28:14 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x4:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  linkea: idx=0 parent=[0x200000402:0x2:0x0] name='d154.conf-sanity-1.dirent'
BLOCKS:
(0):24266
TOTAL: 1

oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  26742   40755 (2)      0      0    4096 15-Dec-2024 06:27 .
  26738   40755 (18)      0      0    4096 15-Dec-2024 06:28 fid:[0x200000402:0x2:0x0] ..
    264  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename1
    265  100644 (1)      0      0       0 15-Dec-2024 06:27 f154.conf-sanity-1.longfilename0

==post-rename .empty should have 'fid:[0x200000402:0x2:0x0] ..'==
oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 26741   Type: directory    Mode:  0755   Flags: 0x0
Generation: 3278234702    Version: 0x00000002:00000003
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x675ebd72:00000000 -- Sun Dec 15 06:28:50 2024
 atime: 0x675ebd4e:2d0fa500 -- Sun Dec 15 06:28:14 2024
 mtime: 0x675ebd23:00000000 -- Sun Dec 15 06:27:31 2024
crtime: 0x675ebd4e:2d0fa500 -- Sun Dec 15 06:28:14 2024
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x5:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  linkea: idx=0 parent=[0x200000402:0x2:0x0] name='d154.conf-sanity-1.empty'
BLOCKS:
(0):24265
TOTAL: 1

oleg422-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg422-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  26741   40755 (2)      0      0    4096 15-Dec-2024 06:27 .
  26738   40755 (18)      0      0    4096 15-Dec-2024 06:28 fid:[0x200000402:0x2:0x0] ..

e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8
oleg422-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg422-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 3)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 24 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 290 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 292 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 294 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] Pass 1: Memory used: 276k/0k (143k/134k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 197.47MB/s
[Thread 0] Scanned group range [0, 3), inodes 414
Pass 2: Checking directory structure
Pass 2: Memory used: 276k/0k (98k/179k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 216.50MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 276k/0k (98k/179k), time:  0.02/ 0.01/ 0.01
Pass 3: Memory used: 276k/0k (96k/181k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 276k/0k (68k/209k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 276k/0k (67k/210k), time:  0.01/ 0.01/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 185.60MB/s

         412 inodes used (0.52%, out of 79992)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
       24569 blocks used (49.14%, out of 50000)
           0 bad blocks
           1 large file

         280 regular files
         122 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         402 files
Memory used: 276k/0k (66k/211k), time:  0.03/ 0.02/ 0.01
I/O read: 1MB, write: 0MB, rate: 33.12MB/s
PASS 154 (214s)
Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f)
oleg422-server: oleg422-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/intel_rapl/refcnt': 'No such device'
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg422-server'
oleg422-server: oleg422-server.virtnet: executing load_modules_local
oleg422-server: Loading modules from /home/green/git/lustre-release/lustre
oleg422-server: detected 4 online CPUs by sysfs
oleg422-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
== conf-sanity-1 test complete, duration 7085 sec ======== 06:30:14 (1734262214)
=== conf-sanity-1: start cleanup 06:30:15 (1734262215) ===
=== conf-sanity-1: finish cleanup 06:30:16 (1734262216) ===