-----============= acceptance-small: conf-sanity ============----- Fri Feb 14 06:44:48 EST 2025
mgs: CentOS Linux release 7.9.2009 (Core)
MGS_OS_VERSION_ID=7
MGS_OS_ID=centos
MGS_OS_VERSION_CODE=117440512
MGS_OS_ID_LIKE=rhel fedora centos
mds1: CentOS Linux release 7.9.2009 (Core)
MDS1_OS_ID_LIKE=rhel fedora centos
MDS1_OS_ID=centos
MDS1_OS_VERSION_ID=7
MDS1_OS_VERSION_CODE=117440512
ost1: CentOS Linux release 7.9.2009 (Core)
OST1_OS_VERSION_CODE=117440512
OST1_OS_VERSION_ID=7
OST1_OS_ID_LIKE=rhel fedora centos
OST1_OS_ID=centos
client: CentOS Linux release 7.9.2009 (Core)
CLIENT_OS_ID=centos
CLIENT_OS_ID_LIKE=rhel fedora centos
CLIENT_OS_VERSION_ID=7
CLIENT_OS_VERSION_CODE=117440512
oleg238-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory
excepting tests: 102 106 115 32newtarball
skipping tests SLOW=no: 45 69 106 111 114
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg238-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server

== conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 06:46:41 (1739533601)
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Change MGS params
max_dirty_mb: 467
new_max_dirty_mb: 457
Waiting 90s for '457'
Updated after 2s: want '457' got '457'
457
Check the value is stored after remount
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff880137679000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff880137679000.idle_timeout=debug
disable quota as required
Change OST params
client_cache_count: 128
new_client_cache_count: 256
Waiting 90s for '256'
Waiting 80s for '256'
Waiting 60s for '256'
Waiting 50s for '256'
Waiting 40s for '256'
Waiting 20s for '256'
Waiting 10s for '256'
Waiting 0s for '256'
Update not seen after 90s: want '256' got '128'
128
 conf-sanity test_76a: @@@@@@ FAIL: error while apply client_cache_count 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7235:error()
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:6962:test_76a()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7608:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7671:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7474:run_test()
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:6976:main()
Dumping lctl log to /tmp/testlogs//conf-sanity.test_76a.*.1739533792.log
rsync: chown "/tmp/testlogs/.conf-sanity.test_76a.debug_log.oleg238-server.1739533792.log.IPPJZV" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.conf-sanity.test_76a.dmesg.oleg238-server.1739533792.log.oRtK7n" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
FAIL 76a (194s)

== conf-sanity test 76b: verify params log setup correctly ========================================================== 06:49:55 (1739533795)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800aaa54000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800aaa54000.idle_timeout=debug
disable quota as required
mgs.MGS.live.params=
fsname: params
flags: 0x20     gen: 3

Secure RPC Config Rules:

imperative_recovery_state:
    state: startup
    nonir_clients: 0
    nidtbl_version: 2
    notify_duration_total: 0.000000000
    notify_duation_max: 0.000000000
    notify_count: 0
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
PASS 76b (111s)

== conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 06:51:46 (1739533906)
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b3adb000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b3adb000.idle_timeout=debug
disable quota as required
Change changelog_mask
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Check the value is stored after mds remount
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
PASS 76c (137s)

== conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 06:54:03 (1739534043)
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800aa92c000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800aa92c000.idle_timeout=debug
disable quota as required
lctl set_param -P llite.*.xattr_cache=0
Waiting 90s for '0'
Updated after 2s: want '0' got '0'
Check llite.*.xattr_cache on client /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Check llite.*.xattr_cache on the new client /mnt/lustre2
mount lustre  on /mnt/lustre2.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre2
umount lustre on /mnt/lustre2.....
Stopping client oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
PASS 76d (86s)

== conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 06:55:29 (1739534129)

 SKIP: conf-sanity test_77 mixed loopback and real device not working
SKIP 77 (2s)

== conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 06:55:32 (1739534132)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format ost1: /dev/mapper/ost1_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=131072
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
create test files
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        83240        1616       73832   3% /mnt/lustre[MDT:0]
lustre-OST0000_UUID       124712        1388      110724   2% /mnt/lustre[OST:0]

filesystem_summary:       124712        1388      110724   2% /mnt/lustre

UUID                      Inodes       IUsed       IFree IUse% Mounted on
lustre-MDT0000_UUID        72000         272       71728   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID        45008         302       44706   1% /mnt/lustre[OST:0]

filesystem_summary:        44978         272       44706   1% /mnt/lustre

1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0797264 s, 13.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0721943 s, 14.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0845367 s, 12.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0656148 s, 16.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0516862 s, 20.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0739125 s, 14.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0578299 s, 18.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.124915 s, 8.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0667166 s, 15.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0566201 s, 18.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0547966 s, 19.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0607441 s, 17.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0567392 s, 18.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0594153 s, 17.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.058961 s, 17.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0595469 s, 17.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0518644 s, 20.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0591595 s, 17.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0589658 s, 17.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0503468 s, 20.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0541587 s, 19.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0492394 s, 21.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0508727 s, 20.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.057063 s, 18.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0465206 s, 22.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0465836 s, 22.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0473954 s, 22.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0495799 s, 21.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0491512 s, 21.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0557724 s, 18.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0512658 s, 20.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0461184 s, 22.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0633624 s, 16.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0504223 s, 20.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0550763 s, 19.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0501736 s, 20.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0520528 s, 20.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0827475 s, 12.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0563722 s, 18.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0500565 s, 20.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0497502 s, 21.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0558162 s, 18.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0475789 s, 22.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0493672 s, 21.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0501675 s, 20.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0521835 s, 20.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0510167 s, 20.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0515615 s, 20.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0534825 s, 19.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0570028 s, 18.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0605793 s, 17.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0576457 s, 18.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0704112 s, 14.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.053062 s, 19.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0624812 s, 16.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0689281 s, 15.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0624683 s, 16.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0556034 s, 18.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0481598 s, 21.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0561997 s, 18.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.131108 s, 8.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0517016 s, 20.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0638281 s, 16.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0570382 s, 18.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0692655 s, 15.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.057137 s, 18.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0611101 s, 17.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0596107 s, 17.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0633347 s, 16.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0578588 s, 18.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0531888 s, 19.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.103741 s, 10.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0679781 s, 15.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0621768 s, 16.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0661469 s, 15.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0589752 s, 17.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0584225 s, 17.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0983241 s, 10.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0659691 s, 15.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0661359 s, 15.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.122358 s, 8.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0646424 s, 16.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0527018 s, 19.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.058478 s, 17.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0584992 s, 17.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.080056 s, 13.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0684472 s, 15.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0686202 s, 15.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0848119 s, 12.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0649653 s, 16.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.060137 s, 17.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0549932 s, 19.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.128921 s, 8.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.085507 s, 12.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0763057 s, 13.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.735276 s, 1.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.057496 s, 18.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.116164 s, 9.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0778794 s, 13.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0872928 s, 12.0 MB/s
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 3)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 24029 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24055 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24056 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24057 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24058 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24059 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48054 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] Pass 1: Memory used: 272k/0k (141k/132k), time:  0.02/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 59.24MB/s
[Thread 0] Scanned group range [0, 3), inodes 373
Pass 2: Checking directory structure
Pass 2: Memory used: 272k/0k (95k/178k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 119.08MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 272k/0k (95k/178k), time:  0.04/ 0.01/ 0.01
Pass 3A: Memory used: 272k/0k (95k/178k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 272k/0k (93k/180k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 6666.67MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 272k/0k (67k/206k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 272k/0k (67k/206k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 323.21MB/s

         372 inodes used (0.52%, out of 72000)
           5 non-contiguous files (1.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
       22546 blocks used (50.10%, out of 45000)
           0 bad blocks
           1 large file

         244 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         362 files
Memory used: 272k/0k (66k/207k), time:  0.06/ 0.01/ 0.01
I/O read: 1MB, write: 1MB, rate: 18.01MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 2)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 22582 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 22583 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] Pass 1: Memory used: 264k/0k (132k/133k), time:  0.01/ 0.01/ 0.00
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 397.05MB/s
[Thread 0] Scanned group range [0, 2), inodes 398
Pass 2: Checking directory structure
Pass 2: Memory used: 264k/0k (86k/179k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 144.07MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 264k/0k (92k/173k), time:  0.03/ 0.01/ 0.01
Pass 3A: Memory used: 264k/0k (92k/173k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 264k/0k (84k/181k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 2932.55MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 264k/0k (65k/200k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 264k/0k (64k/201k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 405.68MB/s

         398 inodes used (0.88%, out of 45008)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 392
       37721 blocks used (83.82%, out of 45000)
           0 bad blocks
           1 large file

         216 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         388 files
Memory used: 264k/0k (64k/201k), time:  0.05/ 0.02/ 0.01
I/O read: 2MB, write: 1MB, rate: 39.26MB/s
oleg238-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/mds1_flakey to 640000 (4k) blocks.
The filesystem on /dev/mapper/mds1_flakey is now 640000 (4k) blocks long.

oleg238-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/ost1_flakey to 1048576 (4k) blocks.
The filesystem on /dev/mapper/ost1_flakey is now 1048576 (4k) blocks long.

e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 2 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 16)
[Thread 1] Scan group range [16, 33)
[Thread 0] jumping to group 0
[Thread 1] jumping to group 16
[Thread 1] group 17 finished
[Thread 1] group 18 finished
[Thread 1] group 19 finished
[Thread 1] group 20 finished
[Thread 1] group 21 finished
[Thread 1] group 22 finished
[Thread 1] group 23 finished
[Thread 1] group 24 finished
[Thread 1] group 25 finished
[Thread 1] group 26 finished
[Thread 1] group 27 finished
[Thread 1] group 28 finished
[Thread 1] group 29 finished
[Thread 1] group 30 finished
[Thread 1] group 31 finished
[Thread 1] group 32 finished
[Thread 1] group 33 finished
[Thread 1] Pass 1: Memory used: 632k/0k (378k/255k), time:  0.00/ 0.00/ 0.00
[Thread 1] Pass 1: I/O read: 1MB, write: 0MB, rate: 257.73MB/s
[Thread 1] Scanned group range [16, 33), inodes 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 24029 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24055 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24056 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24057 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24058 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24059 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48054 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] Pass 1: Memory used: 688k/0k (354k/334k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 87.79MB/s
[Thread 0] Scanned group range [0, 16), inodes 373
Pass 2: Checking directory structure
Pass 2: Memory used: 632k/0k (199k/434k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 68.90MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 632k/0k (200k/433k), time:  0.09/ 0.04/ 0.01
Pass 3A: Memory used: 632k/0k (200k/433k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 632k/0k (198k/435k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 1141.55MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 632k/0k (71k/562k), time:  0.03/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 632k/0k (70k/563k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 257.86MB/s

         372 inodes used (0.05%, out of 792000)
           5 non-contiguous files (1.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
      202726 blocks used (31.68%, out of 640000)
           0 bad blocks
           1 large file

         244 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         362 files
Memory used: 632k/0k (69k/564k), time:  0.13/ 0.07/ 0.01
I/O read: 1MB, write: 1MB, rate: 7.78MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 32)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 22582 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 22583 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] group 19 finished
[Thread 0] group 20 finished
[Thread 0] group 21 finished
[Thread 0] group 22 finished
[Thread 0] group 23 finished
[Thread 0] group 24 finished
[Thread 0] group 25 finished
[Thread 0] group 26 finished
[Thread 0] group 27 finished
[Thread 0] group 28 finished
[Thread 0] group 29 finished
[Thread 0] group 30 finished
[Thread 0] group 31 finished
[Thread 0] group 32 finished
[Thread 0] Pass 1: Memory used: 468k/0k (344k/125k), time:  0.01/ 0.01/ 0.01
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 368.95MB/s
[Thread 0] Scanned group range [0, 32), inodes 398
Pass 2: Checking directory structure
Pass 2: Memory used: 680k/0k (298k/383k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 295.25MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 680k/0k (298k/383k), time:  0.04/ 0.02/ 0.01
Pass 3A: Memory used: 680k/0k (298k/383k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 680k/0k (296k/385k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 5847.95MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 564k/0k (66k/499k), time:  0.02/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 564k/0k (65k/500k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 216.12MB/s

         398 inodes used (0.06%, out of 720128)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 392
      128315 blocks used (12.24%, out of 1048576)
           0 bad blocks
           1 large file

         216 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         388 files
Memory used: 564k/0k (64k/501k), time:  0.07/ 0.04/ 0.01
I/O read: 2MB, write: 1MB, rate: 27.41MB/s
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=131072
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
check files after expanding the MDT and OST filesystems
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-1 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-1 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-2 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-2 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-3 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-3 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-4 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-4 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-5 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-5 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-6 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-6 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-7 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-7 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-8 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-8 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-9 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-9 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-10 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-10 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-11 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-11 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-12 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-12 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-13 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-13 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-14 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-14 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-15 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-15 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-16 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-16 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-17 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-17 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-18 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-18 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-19 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-19 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-20 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-20 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-21 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-21 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-22 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-22 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-23 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-23 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-24 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-24 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-25 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-25 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-26 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-26 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-27 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-27 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-28 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-28 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-29 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-29 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-30 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-30 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-31 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-31 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-32 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-32 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-33 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-33 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-34 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-34 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-35 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-35 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-36 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-36 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-37 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-37 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-38 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-38 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-39 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-39 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-40 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-40 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-41 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-41 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-42 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-42 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-43 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-43 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-44 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-44 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-45 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-45 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-46 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-46 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-47 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-47 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-48 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-48 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-49 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-49 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-50 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-50 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-51 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-51 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-52 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-52 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-53 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-53 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-54 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-54 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-55 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-55 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-56 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-56 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-57 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-57 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-58 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-58 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-59 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-59 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-60 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-60 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-61 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-61 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-62 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-62 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-63 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-63 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-64 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-64 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-65 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-65 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-66 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-66 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-67 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-67 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-68 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-68 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-69 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-69 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-70 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-70 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-71 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-71 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-72 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-72 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-73 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-73 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-74 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-74 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-75 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-75 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-76 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-76 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-77 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-77 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-78 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-78 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-79 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-79 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-80 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-80 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-81 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-81 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-82 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-82 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-83 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-83 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-84 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-84 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-85 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-85 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-86 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-86 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-87 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-87 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-88 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-88 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-89 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-89 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-90 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-90 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-91 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-91 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-92 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-92 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-93 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-93 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-94 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-94 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-95 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-95 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-96 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-96 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-97 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-97 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-98 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-98 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-99 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-99 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-100 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-100 has size 1048576 OK
create more files after expanding the MDT and OST filesystems
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0682433 s, 15.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0631802 s, 16.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0730564 s, 14.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0751558 s, 14.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0586706 s, 17.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0640344 s, 16.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0830922 s, 12.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.136219 s, 7.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.106456 s, 9.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0719653 s, 14.6 MB/s
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 2 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 16)
[Thread 1] Scan group range [16, 33)
[Thread 1] jumping to group 16
[Thread 0] jumping to group 0
[Thread 1] group 17 finished
[Thread 1] group 18 finished
[Thread 1] group 19 finished
[Thread 1] group 20 finished
[Thread 1] group 21 finished
[Thread 1] group 22 finished
[Thread 1] group 23 finished
[Thread 1] group 24 finished
[Thread 1] group 25 finished
[Thread 1] group 26 finished
[Thread 1] group 27 finished
[Thread 1] group 28 finished
[Thread 1] group 29 finished
[Thread 1] group 30 finished
[Thread 1] group 31 finished
[Thread 1] group 32 finished
[Thread 1] group 33 finished
[Thread 1] Pass 1: Memory used: 632k/0k (381k/252k), time:  0.00/ 0.00/ 0.00
[Thread 1] Pass 1: I/O read: 1MB, write: 0MB, rate: 995.02MB/s
[Thread 1] Scanned group range [16, 33), inodes 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 24029 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24055 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24056 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24057 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24058 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24059 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48054 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] Pass 1: Memory used: 688k/0k (355k/334k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 87.44MB/s
[Thread 0] Scanned group range [0, 16), inodes 383
Pass 2: Checking directory structure
Pass 2: Memory used: 632k/0k (199k/433k), time:  0.01/ 0.01/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 69.81MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 632k/0k (200k/433k), time:  0.07/ 0.04/ 0.01
Pass 3A: Memory used: 632k/0k (200k/433k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 632k/0k (198k/435k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 4975.12MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 632k/0k (72k/561k), time:  0.03/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 632k/0k (70k/563k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 170.85MB/s

         382 inodes used (0.05%, out of 792000)
           5 non-contiguous files (1.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
      202726 blocks used (31.68%, out of 640000)
           0 bad blocks
           1 large file

         254 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         372 files
Memory used: 632k/0k (69k/564k), time:  0.12/ 0.07/ 0.01
I/O read: 1MB, write: 1MB, rate: 8.69MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 32)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 22582 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 22583 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] group 19 finished
[Thread 0] group 20 finished
[Thread 0] group 21 finished
[Thread 0] group 22 finished
[Thread 0] group 23 finished
[Thread 0] group 24 finished
[Thread 0] group 25 finished
[Thread 0] group 26 finished
[Thread 0] group 27 finished
[Thread 0] group 28 finished
[Thread 0] group 29 finished
[Thread 0] group 30 finished
[Thread 0] group 31 finished
[Thread 0] group 32 finished
[Thread 0] Pass 1: Memory used: 468k/0k (344k/125k), time:  0.02/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 278.30MB/s
[Thread 0] Scanned group range [0, 32), inodes 402
Pass 2: Checking directory structure
Pass 2: Memory used: 680k/0k (298k/383k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 125.74MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 680k/0k (298k/383k), time:  0.06/ 0.02/ 0.01
Pass 3A: Memory used: 680k/0k (299k/382k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 680k/0k (296k/385k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 484.73MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 564k/0k (66k/499k), time:  0.03/ 0.02/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 564k/0k (65k/500k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 113.01MB/s

         402 inodes used (0.06%, out of 720128)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 396
      130875 blocks used (12.48%, out of 1048576)
           0 bad blocks
           1 large file

         220 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         392 files
Memory used: 564k/0k (64k/501k), time:  0.10/ 0.05/ 0.01
I/O read: 2MB, write: 1MB, rate: 19.37MB/s
oleg238-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/mds1_flakey to 377837 (4k) blocks.
The filesystem on /dev/mapper/mds1_flakey is now 377837 (4k) blocks long.

oleg238-server: resize2fs 1.46.2.wc5 (26-Mar-2022)
Resizing the filesystem on /dev/mapper/ost1_flakey to 591846 (4k) blocks.
The filesystem on /dev/mapper/ost1_flakey is now 589824 (4k) blocks long.

e2fsck -d -v -t -t -f -y /dev/mapper/mds1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 20)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 24029 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24055 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24056 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24057 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24058 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 24059 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 48050 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48051 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48052 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48053 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 48054 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] group 19 finished
[Thread 0] group 20 finished
[Thread 0] Pass 1: Memory used: 400k/0k (269k/132k), time:  0.02/ 0.01/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 65.96MB/s
[Thread 0] Scanned group range [0, 20), inodes 383
Pass 2: Checking directory structure
Pass 2: Memory used: 576k/0k (224k/353k), time:  0.01/ 0.01/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 94.30MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 576k/0k (224k/353k), time:  0.05/ 0.03/ 0.01
Pass 3A: Memory used: 576k/0k (224k/353k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 576k/0k (222k/355k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 2500.00MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 500k/0k (68k/433k), time:  0.03/ 0.01/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 500k/0k (67k/434k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 250.50MB/s

         382 inodes used (0.08%, out of 480000)
           5 non-contiguous files (1.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
      124660 blocks used (32.99%, out of 377837)
           0 bad blocks
           1 large file

         254 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         372 files
Memory used: 500k/0k (66k/435k), time:  0.08/ 0.05/ 0.01
I/O read: 1MB, write: 1MB, rate: 11.82MB/s
e2fsck -d -v -t -t -f -y /dev/mapper/ost1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 18)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 22582 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 22583 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] group 4 finished
[Thread 0] group 5 finished
[Thread 0] group 6 finished
[Thread 0] group 7 finished
[Thread 0] group 8 finished
[Thread 0] group 9 finished
[Thread 0] group 10 finished
[Thread 0] group 11 finished
[Thread 0] group 12 finished
[Thread 0] group 13 finished
[Thread 0] group 14 finished
[Thread 0] group 15 finished
[Thread 0] group 16 finished
[Thread 0] group 17 finished
[Thread 0] group 18 finished
[Thread 0] Pass 1: Memory used: 372k/0k (245k/128k), time:  0.02/ 0.00/ 0.01
[Thread 0] Pass 1: I/O read: 5MB, write: 0MB, rate: 311.93MB/s
[Thread 0] Scanned group range [0, 18), inodes 402
Pass 2: Checking directory structure
Pass 2: Memory used: 532k/0k (199k/334k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 137.19MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 532k/0k (200k/333k), time:  0.04/ 0.02/ 0.01
Pass 3A: Memory used: 532k/0k (200k/333k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 532k/0k (197k/336k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 1926.78MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 468k/0k (66k/403k), time:  0.02/ 0.01/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 468k/0k (65k/404k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 192.46MB/s

         402 inodes used (0.10%, out of 405072)
           2 non-contiguous files (0.5%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 396
       89417 blocks used (15.16%, out of 589824)
           0 bad blocks
           1 large file

         220 regular files
         172 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         392 files
Memory used: 468k/0k (64k/405k), time:  0.07/ 0.03/ 0.01
I/O read: 2MB, write: 1MB, rate: 28.25MB/s
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=131072
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
check files after shrinking the MDT and OST filesystems
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-1 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-1 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-2 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-2 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-3 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-3 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-4 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-4 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-5 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-5 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-6 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-6 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-7 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-7 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-8 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-8 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-9 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-9 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-10 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-10 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-11 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-11 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-12 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-12 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-13 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-13 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-14 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-14 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-15 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-15 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-16 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-16 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-17 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-17 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-18 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-18 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-19 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-19 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-20 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-20 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-21 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-21 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-22 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-22 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-23 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-23 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-24 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-24 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-25 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-25 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-26 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-26 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-27 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-27 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-28 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-28 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-29 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-29 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-30 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-30 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-31 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-31 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-32 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-32 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-33 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-33 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-34 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-34 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-35 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-35 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-36 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-36 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-37 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-37 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-38 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-38 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-39 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-39 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-40 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-40 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-41 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-41 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-42 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-42 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-43 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-43 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-44 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-44 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-45 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-45 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-46 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-46 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-47 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-47 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-48 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-48 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-49 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-49 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-50 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-50 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-51 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-51 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-52 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-52 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-53 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-53 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-54 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-54 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-55 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-55 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-56 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-56 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-57 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-57 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-58 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-58 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-59 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-59 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-60 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-60 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-61 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-61 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-62 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-62 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-63 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-63 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-64 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-64 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-65 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-65 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-66 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-66 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-67 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-67 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-68 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-68 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-69 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-69 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-70 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-70 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-71 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-71 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-72 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-72 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-73 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-73 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-74 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-74 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-75 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-75 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-76 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-76 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-77 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-77 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-78 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-78 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-79 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-79 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-80 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-80 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-81 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-81 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-82 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-82 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-83 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-83 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-84 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-84 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-85 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-85 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-86 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-86 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-87 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-87 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-88 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-88 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-89 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-89 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-90 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-90 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-91 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-91 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-92 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-92 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-93 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-93 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-94 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-94 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-95 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-95 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-96 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-96 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-97 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-97 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-98 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-98 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-99 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-99 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-100 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-100 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-101 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-101 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-102 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-102 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-103 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-103 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-104 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-104 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-105 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-105 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-106 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-106 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-107 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-107 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-108 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-108 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-109 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-109 has size 1048576 OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-110 has type file OK
/mnt/lustre/d78.conf-sanity/f78.conf-sanity-110 has size 1048576 OK
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 78 (235s)

== conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 06:59:27 (1739534367)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: 
oleg238-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg238-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 22
oleg238-server: 
oleg238-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg238-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 22
oleg238-server: 
oleg238-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg238-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 22
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 79 (69s)

== conf-sanity test 80: mgc import reconnect race ======== 07:00:36 (1739534436)
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
fail_val=10
fail_loc=0x906
fail_val=10
fail_loc=0x906
start ost2 service on oleg238-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
fail_loc=0
stop ost2 service on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 80 (72s)

== conf-sanity test 81: sparse OST indexing ============== 07:01:48 (1739534508)

 SKIP: conf-sanity test_81 needs >= 3 OSTs
SKIP 81 (1s)

== conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 07:01:49 (1739534509)

 SKIP: conf-sanity test_82a needs >= 3 OSTs
SKIP 82a (1s)

== conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 07:01:50 (1739534510)

 SKIP: conf-sanity test_82b needs >= 4 OSTs
SKIP 82b (1s)

== conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 07:01:51 (1739534511)
mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem
mnt_opts -o loop
run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1
oleg238-server: oleg238-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no
oleg238-server: 
oleg238-server: llverfs: write /mnt/lustre-ost1/llverfs_dir00142/file000@0+1048576 short: 368640 written
oleg238-server: Timestamp: 1739534515
oleg238-server: dirs: 147, fs blocks: 37602
oleg238-server: write_done: /mnt/lustre-ost1/llverfs_dir00142/file000, current: 255.541 MB/s, overall: 255.541 MB/s, ETA: 0:00:00
oleg238-server: 
oleg238-server: read_done: /mnt/lustre-ost1/llverfs_dir00141/file000, current: 3392.26 MB/s, overall: 3392.26 MB/s, ETA: 0:00:00
oleg238-server: 
unmount the OST /dev/mapper/ost1_flakey
Stopping /mnt/lustre-ost1 (opts:) on oleg238-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg238-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: No space left on device
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 28
oleg238-server: error: set_param: param_path 'seq/cli-lustre': No such file or directory
oleg238-server: error: set_param: setting 'seq/cli-lustre'='OST0000-super.width=65536': No such file or directory
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 28
string err 
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 83 (76s)

== conf-sanity test 84: check recovery_hard_time ========= 07:03:07 (1739534587)
start mds service on oleg238-server
start mds service on oleg238-server
Starting mds1: -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec
start ost2 service on oleg238-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
recovery_time=60, timeout=20, wrap_up=5
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
mount lustre  on /mnt/lustre2.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre2
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        95248        1668       84924   2% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        95248        1532       85060   2% /mnt/lustre[MDT:1]
lustre-OST0000_UUID       142216        1524      126692   2% /mnt/lustre[OST:0]
lustre-OST0001_UUID       142216        1524      126692   2% /mnt/lustre[OST:1]

filesystem_summary:       284432        3048      253384   2% /mnt/lustre

total: 1000 open/close in 4.03 seconds: 247.93 ops/second
fail_loc=0x20000709
fail_val=5
Failing mds1 on oleg238-server
Stopping /mnt/lustre-mds1 (opts:) on oleg238-server
07:03:49 (1739534629) shut down
facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server
Failover mds1 to oleg238-server
e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 1 instead
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 3)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 159 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 26697 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 53372 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53373 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53374 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53375 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53376 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53377 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53378 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53379 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53380 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53381 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53382 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] Pass 1: Memory used: 264k/0k (140k/125k), time:  0.00/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 253.55MB/s
[Thread 0] Scanned group range [0, 3), inodes 277
Pass 2: Checking directory structure
Pass 2: Memory used: 264k/0k (97k/168k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 147.75MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 264k/0k (97k/168k), time:  0.02/ 0.01/ 0.00
Pass 3: Memory used: 264k/0k (95k/170k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 264k/0k (67k/198k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Free blocks count wrong (25455, counted=25443).
Fix? no

Free inodes count wrong (79719, counted=79715).
Fix? no

Pass 5: Memory used: 264k/0k (67k/198k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 287.03MB/s

         273 inodes used (0.34%, out of 79992)
           5 non-contiguous files (1.8%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
       24545 blocks used (49.09%, out of 50000)
           0 bad blocks
           1 large file

         150 regular files
         117 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         267 files
Memory used: 264k/0k (66k/199k), time:  0.03/ 0.01/ 0.00
I/O read: 1MB, write: 0MB, rate: 35.10MB/s
mount facets: mds1
Starting mds1: -o localrecov -o recovery_time_hard=60,recovery_time_soft=60  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
07:04:04 (1739534644) targets are mounted
07:04:04 (1739534644) facet_failover done
oleg238-client: error: invalid path '/mnt/lustre': Input/output error
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 5
recovery status
status: COMPLETE
recovery_start: 1739534645
recovery_duration: 60
completed_clients: 2/3
replayed_requests: 152
last_transno: 8589934744
VBR: DISABLED
IR: DISABLED
fail_loc=0
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
umount lustre on /mnt/lustre2.....
Stopping client oleg238-client.virtnet /mnt/lustre2 (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop ost2 service on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 84 (146s)

== conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 07:05:33 (1739534733)
fail_loc=0x197
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
PASS 85 (63s)

== conf-sanity test 86: Replacing mkfs.lustre -G option == 07:06:37 (1739534797)
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps
params: --mgsnode=192.168.202.138@tcp --fsname=lustre --ost 			--index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-G 1024 -b 4096 -O flex_bg
 -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey 
Failing mds1 on oleg238-server
07:06:39 (1739534799) shut down
facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server
Failover mds1 to oleg238-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
07:06:54 (1739534814) targets are mounted
07:06:54 (1739534814) facet_failover done
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 95
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 95
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST0000
	kilobytes     200000
	options       -G 1024 -b 4096   -I 512 -q -O flex_bg,uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -G 1024 -b 4096   -I 512 -q -O flex_bg,uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 86 (104s)

== conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 07:08:20 (1739534900)
Estimate: at most 353-byte space left in inode.
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
setup single mount lustre success
4 -rw-r--r-- 1 root root 67108865 Feb 14 07:09 /mnt/lustre-mds1/ROOT/f87.conf-sanity
Verified: at most 353-byte space left in inode.
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
PASS 87 (92s)

== conf-sanity test 88: check the default mount options can be overridden ========================================================== 07:09:52 (1739534992)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata
Persistent mount opts: user_xattr,errors=remount-ro
Persistent mount opts: user_xattr,errors=remount-ro

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=panic
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -b 4096  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata
Persistent mount opts: user_xattr,errors=panic
Persistent mount opts: user_xattr,errors=panic
PASS 88 (15s)

== conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 07:10:07 (1739535007)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
tunefs --param failover.node=192.0.2.254@tcp0
tunefs --param failover.node=192.0.2.255@tcp0
tunefs --erase-param failover.node
tunefs --erase-params
tunefs --param failover.node=192.0.2.254@tcp0 --erase-params
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 89 (85s)

== conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 07:11:32 (1739535092)
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 8 files ...
fail_loc=0x159
launch 6 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-7 has perms 0600 OK
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90a.conf-sanity/file-8 has perms 0644 OK
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 90a (92s)

== conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ========================================================== 07:13:04 (1739535184)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff8800b3adf800.max_mod_rpcs_in_flight=1
max_mod_rpcs_in_flight set to 1
creating 2 files ...
fail_loc=0x159
launch 0 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-1 has perms 0600 OK
fail_loc=0x159
launch 1 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity1/file-2 has perms 0644 OK
mdc.lustre-MDT0001-mdc-ffff8800b3adf800.max_mod_rpcs_in_flight=5
max_mod_rpcs_in_flight set to 5
creating 6 files ...
fail_loc=0x159
launch 4 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-5 has perms 0600 OK
fail_loc=0x159
launch 5 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity2/file-6 has perms 0644 OK
mdt_max_mod_rpcs_in_flight is 8
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff8800aaa54000.max_rpcs_in_flight=17
mdc.lustre-MDT0000-mdc-ffff8800aaa54000.max_mod_rpcs_in_flight=16
max_mod_rpcs_in_flight set to 16
creating 17 files ...
fail_loc=0x159
launch 15 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-16 has perms 0600 OK
fail_loc=0x159
launch 16 chmod in parallel ...
fail_loc=0
launch 1 additional chmod in parallel ...
/mnt/lustre/d90b.conf-sanity3/file-17 has perms 0644 OK
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 90b (158s)

== conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ========================================================== 07:15:42 (1739535342)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
max_rpcs_in_flight is 8
MDC max_mod_rpcs_in_flight is 7
mdt_max_mod_rpcs_in_flight is 8
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mdc.lustre-MDT0000-mdc-ffff8800aa92e000.max_mod_rpcs_in_flight=8
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
mdc.lustre-MDT0000-mdc-ffff8800b6391800.max_rpcs_in_flight=10
error: set_param: setting /sys/fs/lustre/mdc/lustre-MDT0000-mdc-ffff8800b6391800/max_mod_rpcs_in_flight=9: Numerical result out of range
error: set_param: setting 'mdc/lustre-MDT0000-mdc-*/max_mod_rpcs_in_flight'='9': Numerical result out of range
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory
the deprecated max_mod_rpcs_per_client 				parameter was involved
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 90c (79s)

== conf-sanity test 90d: check one close RPC is allowed above max_mod_rpcs_in_flight ========================================================== 07:17:01 (1739535421)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
max_mod_rcps_in_flight is 7
creating 7 files ...
multiop /mnt/lustre/d90d.conf-sanity/file-close vO_c
TMPPIPE=/tmp/multiop_open_wait_pipe.7624
fail_loc=0x159
launch 7 chmod in parallel ...
fail_loc=0
launch 1 additional close in parallel ...
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 90d (117s)

== conf-sanity test 91: evict-by-nid support ============= 07:18:58 (1739535538)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
setup single mount lustre success
list nids on mdt:
mdt.lustre-MDT0000.exports.0@lo
mdt.lustre-MDT0000.exports.192.168.202.38@tcp
mdt.lustre-MDT0000.exports.clear
mdt.lustre-MDT0001.exports.0@lo
mdt.lustre-MDT0001.exports.192.168.202.38@tcp
mdt.lustre-MDT0001.exports.clear
uuid from 192\.168\.202\.38@tcp:
mdt.lustre-MDT0000.exports.192.168.202.38@tcp.uuid=692a7c05-fe9a-4f40-a5b6-85ef0bbd1165
mdt.lustre-MDT0001.exports.192.168.202.38@tcp.uuid=692a7c05-fe9a-4f40-a5b6-85ef0bbd1165
manual umount lustre on /mnt/lustre....
evict 192\.168\.202\.38@tcp
oleg238-server: error: read_param: '/proc/fs/lustre/mdt/lustre-MDT0000/exports/192.168.202.38@tcp/uuid': No such device
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 19
oleg238-server: error: read_param: '/proc/fs/lustre/obdfilter/lustre-OST0000/exports/192.168.202.38@tcp/uuid': No such device
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 19
oleg238-server: error: read_param: '/proc/fs/lustre/mdt/lustre-MDT0000/exports/192.168.202.38@tcp/uuid': No such device
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 19
oleg238-server: error: read_param: '/proc/fs/lustre/obdfilter/lustre-OST0000/exports/192.168.202.38@tcp/uuid': No such device
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 19
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 91 (107s)

== conf-sanity test 92: ldev returns MGS NID correctly in command substitution ========================================================== 07:20:45 (1739535645)
Host is oleg238-client.virtnet
----- /tmp/ldev.conf -----
oleg238-server	-	lustre-MGS0000	/dev/mapper/mds1_flakey
oleg238-server	-	lustre-OST0000	/dev/mapper/ost1_flakey
oleg238-server	-	lustre-OST0001	/dev/mapper/ost2_flakey
oleg238-server	-	lustre-MDT0000	/dev/mapper/mds1_flakey
oleg238-server	-	lustre-MDT0001	/dev/mapper/mds2_flakey
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg238-server	192.168.202.138@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.202.138@tcp
lustre-MDT0001: 192.168.202.138@tcp
lustre-MDT0000: 192.168.202.138@tcp
lustre-MGS0000: 192.168.202.138@tcp
lustre-OST0001: 192.168.202.138@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0001: 192.168.202.138@tcp
lustre-MDT0000: 192.168.202.138@tcp
lustre-MGS0000: 192.168.202.138@tcp
lustre-MDT0001: 192.168.202.138@tcp
lustre-OST0000: 192.168.202.138@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.202.138@tcp
lustre-MGS0000: 192.168.202.138@tcp
lustre-OST0001: 192.168.202.138@tcp
lustre-MDT0000: 192.168.202.138@tcp
lustre-MDT0001: 192.168.202.138@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-OST0000: 192.168.202.138@tcp
lustre-OST0001: 192.168.202.138@tcp
lustre-MGS0000: 192.168.202.138@tcp
lustre-MDT0000: 192.168.202.138@tcp
lustre-MDT0001: 192.168.202.138@tcp
--- END OF LDEV OUTPUT ---
-- START OF LDEV OUTPUT --
lustre-MGS0000: 192.168.202.138@tcp
lustre-OST0000: 192.168.202.138@tcp
lustre-OST0001: 192.168.202.138@tcp
lustre-MDT0000: 192.168.202.138@tcp
lustre-MDT0001: 192.168.202.138@tcp
--- END OF LDEV OUTPUT ---
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 92 (4s)

== conf-sanity test 93: register mulitple MDT at the same time ========================================================== 07:20:49 (1739535649)

 SKIP: conf-sanity test_93 needs >= 3 MDTs
SKIP 93 (1s)

== conf-sanity test 94: ldev outputs correct labels for file system name query ========================================================== 07:20:50 (1739535650)
----- /tmp/ldev.conf -----
oleg238-server	-	lustre-MGS0000	/dev/mapper/mds1_flakey
oleg238-server	-	lustre-OST0000	/dev/mapper/ost1_flakey
oleg238-server	-	lustre-OST0001	/dev/mapper/ost2_flakey
oleg238-server	-	lustre-MDT0000	/dev/mapper/mds1_flakey
oleg238-server	-	lustre-MDT0001	/dev/mapper/mds2_flakey
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg238-server	192.168.202.138@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
lustre-MDT0000
lustre-MDT0001
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
lustre-MDT0001
lustre-MGS0000
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 94 (3s)

== conf-sanity test 95: ldev should only allow one label filter ========================================================== 07:20:53 (1739535653)
----- /tmp/ldev.conf -----
oleg238-server	-	lustre-MGS0000	/dev/mapper/mds1_flakey
oleg238-server	-	lustre-OST0000	/dev/mapper/ost1_flakey
oleg238-server	-	lustre-OST0001	/dev/mapper/ost2_flakey
oleg238-server	-	lustre-MDT0000	/dev/mapper/mds1_flakey
oleg238-server	-	lustre-MDT0001	/dev/mapper/mds2_flakey
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg238-server	192.168.202.138@tcp
--- END /tmp/nids ---
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 95 (4s)

== conf-sanity test 96: ldev returns hostname and backend fs correctly in command sub ========================================================== 07:20:57 (1739535657)
----- /tmp/ldev.conf -----
oleg238-server	-	lustre-MGS0000	/dev/mapper/mds1_flakey
oleg238-server	-	lustre-OST0000	/dev/mapper/ost1_flakey
oleg238-server	-	lustre-OST0001	/dev/mapper/ost2_flakey
oleg238-server	-	lustre-MDT0000	/dev/mapper/mds1_flakey
oleg238-server	-	lustre-MDT0001	/dev/mapper/mds2_flakey
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg238-server	192.168.202.138@tcp
--- END /tmp/nids ---
-- START OF LDEV OUTPUT --
oleg238-server-ldiskfs
oleg238-server-ldiskfs
oleg238-server-ldiskfs
oleg238-server-ldiskfs
oleg238-server-ldiskfs
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
oleg238-server-ldiskfs
oleg238-server-ldiskfs
oleg238-server-ldiskfs
oleg238-server-ldiskfs
oleg238-server-ldiskfs
--- END OF EXPECTED OUTPUT ---
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 96 (4s)

== conf-sanity test 97: ldev returns correct ouput when querying based on role ========================================================== 07:21:01 (1739535661)
----- /tmp/ldev.conf -----
oleg238-server	-	lustre-MGS0000	/dev/mapper/mds1_flakey
oleg238-server	-	lustre-OST0000	/dev/mapper/ost1_flakey
oleg238-server	-	lustre-OST0001	/dev/mapper/ost2_flakey
oleg238-server	-	lustre-MDT0000	/dev/mapper/mds1_flakey
oleg238-server	-	lustre-MDT0001	/dev/mapper/mds2_flakey
--- END /tmp/ldev.conf ---
----- /tmp/nids -----
oleg238-server	192.168.202.138@tcp
--- END /tmp/nids ---

MDT role
-- START OF LDEV OUTPUT --
lustre-MDT0000
lustre-MDT0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MDT0000
lustre-MDT0001
--- END OF EXPECTED OUTPUT ---

OST role
-- START OF LDEV OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-OST0000
lustre-OST0001
--- END OF EXPECTED OUTPUT ---

MGS role
-- START OF LDEV OUTPUT --
lustre-MGS0000
--- END OF LDEV OUTPUT ---
-- START OF EXPECTED OUTPUT --
lustre-MGS0000
--- END OF EXPECTED OUTPUT ---
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 97 (4s)

== conf-sanity test 98: Buffer-overflow check while parsing mount_opts ========================================================== 07:21:06 (1739535666)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
setup single mount lustre success
arg[0] = mount.lustre
arg[1] = -o
arg[2] = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
arg[3] = -v
arg[4] = 192.168.202.138@tcp:/lustre
arg[5] = /mnt/lustre
source = 192.168.202.138@tcp:/lustre (192.168.202.138@tcp:/lustre), target = /mnt/lustre
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr
options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 98 (74s)

== conf-sanity test 99: Adding meta_bg option ============ 07:22:19 (1739535739)
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps
params: --mgsnode=192.168.202.138@tcp --fsname=lustre --ost 			--index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000  		--mkfsoptions=\"-O ^resize_inode,meta_bg -b 4096 -E lazy_itable_init\" 		--reformat /dev/mapper/ost1_flakey 
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

device size = 4096MB
formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey
	target name   lustre:OST0000
	kilobytes     200000
	options        -b 4096  -I 512 -q -O ^resize_inode,meta_bg,uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000  -b 4096  -I 512 -q -O ^resize_inode,meta_bg,uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k
Writing CONFIGS/mountdata
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps
Filesystem features:      has_journal ext_attr dir_index filetype meta_bg extent flex_bg large_dir sparse_super large_file huge_file uninit_bg dir_nlink quota project
PASS 99 (16s)

== conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 07:22:35 (1739535755)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
setup single mount lustre success
lustre-MDT0000:
lustre-OST0000:
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 100 (79s)

== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:23:54 (1739535834)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
 - open/close 195 (time 1739535890.96 total 10.00 last 19.50)
 - open/close 395 (time 1739535901.86 total 20.90 last 18.35)
 - open/close 589 (time 1739535912.08 total 31.12 last 18.99)
 - open/close 881 (time 1739535922.71 total 41.75 last 27.45)
 - open/close 1184 (time 1739535934.44 total 53.48 last 25.85)
 - open/close 1395 (time 1739535945.20 total 64.24 last 19.60)
 - open/close 1575 (time 1739535955.64 total 74.68 last 17.24)
 - open/close 1921 (time 1739535965.70 total 84.75 last 34.39)
 - open/close 2156 (time 1739535975.78 total 94.82 last 23.33)
 - open/close 2474 (time 1739535986.74 total 105.78 last 29.00)
 - open/close 2799 (time 1739535997.57 total 116.61 last 30.02)
 - open/close 3034 (time 1739536007.57 total 126.62 last 23.49)
 - open/close 3171 (time 1739536017.64 total 136.68 last 13.61)
 - open/close 3451 (time 1739536027.67 total 146.71 last 27.92)
 - open/close 3759 (time 1739536038.08 total 157.12 last 29.58)
 - open/close 4065 (time 1739536048.43 total 167.47 last 29.58)
 - open/close 4336 (time 1739536058.75 total 177.79 last 26.24)
 - open/close 4568 (time 1739536069.81 total 188.86 last 20.98)
 - open/close 4724 (time 1739536080.16 total 199.20 last 15.08)
 - open/close 4909 (time 1739536090.24 total 209.29 last 18.34)
 - open/close 5298 (time 1739536101.02 total 220.06 last 36.11)
 - open/close 5653 (time 1739536111.86 total 230.91 last 32.72)
 - open/close 5853 (time 1739536122.67 total 241.71 last 18.51)
 - open/close 6165 (time 1739536133.08 total 252.13 last 29.97)
 - open/close 6452 (time 1739536143.80 total 262.84 last 26.78)
 - open/close 7053 (time 1739536154.21 total 273.25 last 57.75)
 - open/close 7258 (time 1739536164.78 total 283.82 last 19.39)
 - open/close 7586 (time 1739536175.60 total 294.64 last 30.31)
 - open/close 7839 (time 1739536186.45 total 305.49 last 23.33)
 - open/close 8098 (time 1739536196.69 total 315.73 last 25.29)
 - open/close 10000 (time 1739536206.21 total 325.25 last 199.75)
 - open/close 13752 (time 1739536216.21 total 335.25 last 375.20)
 - open/close 16877 (time 1739536226.21 total 345.25 last 312.47)
 - open/close 20000 (time 1739536235.37 total 354.42 last 340.85)
 - open/close 24641 (time 1739536245.38 total 364.42 last 464.09)
 - open/close 29370 (time 1739536255.38 total 374.42 last 472.81)
 - open/close 33930 (time 1739536265.38 total 384.42 last 455.99)
 - open/close 39666 (time 1739536275.38 total 394.42 last 573.54)
 - open/close 45190 (time 1739536285.38 total 404.42 last 552.37)
open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device
total: 49632 open/close in 413.41 seconds: 120.05 ops/second
 - unlinked 0 (time 1739536295 ; total 0 ; last 0)
 - unlinked 10000 (time 1739536305 ; total 10 ; last 10)
 - unlinked 20000 (time 1739536315 ; total 20 ; last 10)
 - unlinked 30000 (time 1739536325 ; total 30 ; last 10)
 - unlinked 40000 (time 1739536335 ; total 40 ; last 10)
unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory
total: 49632 unlinks in 52 seconds: 954.461548 unlinks/second
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 101a (534s)

== conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 07:32:48 (1739536368)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
fail_loc=0x80002107
fail_val=20
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec
oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b38d0000.ost_server_uuid 50
osc.lustre-OST0000-osc-ffff8800b38d0000.ost_server_uuid in FULL state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 101b (97s)

 SKIP: conf-sanity test_102 skipping excluded test 102

== conf-sanity test 103: rename filesystem name ========== 07:34:25 (1739536465)
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800aaa54800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800aaa54800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 2s: want 'procname_uid' got 'procname_uid'
disable quota as required
oleg238-server: Pool lustre.pool1 created
oleg238-server: Pool lustre.lustre created
oleg238-server: OST lustre-OST0000_UUID added to pool lustre.lustre
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
rename lustre to mylustre
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

cmd: tune2fs -f -L 'mylustre-MDT0000' '/dev/mapper/mds1_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     mylustre-MDT0001
Index:      1
Lustre FS:  mylustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

cmd: tune2fs -f -L 'mylustre-MDT0001' '/dev/mapper/mds2_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

cmd: tune2fs -f -L 'mylustre-OST0000' '/dev/mapper/ost1_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

cmd: tune2fs -f -L 'mylustre-OST0001' '/dev/mapper/ost2_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started mylustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started mylustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-mylustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started mylustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-mylustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started mylustre-OST0001
mount mylustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/mylustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/mylustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/mylustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.mylustre-OST0000-osc-ffff88012a5e6800.idle_timeout=debug
osc.mylustre-OST0001-osc-ffff88012a5e6800.idle_timeout=debug
disable quota as required
  File: '/mnt/lustre/d103.conf-sanity/test-framework.sh'
  Size: 308604    	Blocks: 608        IO Block: 4194304 regular file
Device: c3aa56ceh/3282720462d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-02-14 07:35:04.000000000 -0500
Modify: 2025-02-14 07:35:04.000000000 -0500
Change: 2025-02-14 07:35:04.000000000 -0500
 Birth: -
Pool: mylustre.pool1
Pool: mylustre.lustre
mylustre-OST0000_UUID
mylustre-OST0000_UUID
oleg238-server: OST mylustre-OST0001_UUID added to pool mylustre.lustre
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
rename mylustre to tfs
checking for existing Lustre data: found

   Read previous values:
Target:     mylustre-MDT0000
Index:      0
Lustre FS:  mylustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

cmd: tune2fs -f -L 'tfs-MDT0000' '/dev/mapper/mds1_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
checking for existing Lustre data: found

   Read previous values:
Target:     mylustre-MDT0001
Index:      1
Lustre FS:  mylustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     tfs-MDT0001
Index:      1
Lustre FS:  tfs
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

cmd: tune2fs -f -L 'tfs-MDT0001' '/dev/mapper/mds2_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
checking for existing Lustre data: found

   Read previous values:
Target:     mylustre-OST0000
Index:      0
Lustre FS:  mylustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

cmd: tune2fs -f -L 'tfs-OST0000' '/dev/mapper/ost1_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
checking for existing Lustre data: found

   Read previous values:
Target:     mylustre-OST0001
Index:      1
Lustre FS:  mylustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

cmd: tune2fs -f -L 'tfs-OST0001' '/dev/mapper/ost2_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started tfs-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started tfs-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-tfs-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started tfs-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-tfs-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started tfs-OST0001
mount tfs  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/tfs /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/tfs /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/tfs on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.tfs-OST0000-osc-ffff8800ad3b5000.idle_timeout=debug
osc.tfs-OST0001-osc-ffff8800ad3b5000.idle_timeout=debug
disable quota as required
  File: '/mnt/lustre/d103.conf-sanity/test-framework.sh'
  Size: 308604    	Blocks: 608        IO Block: 4194304 regular file
Device: 32e2fa5ah/853736026d	Inode: 144115305935798275  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-02-14 07:35:04.000000000 -0500
Modify: 2025-02-14 07:35:04.000000000 -0500
Change: 2025-02-14 07:35:04.000000000 -0500
 Birth: -
Pool: tfs.pool1
Pool: tfs.lustre
tfs-OST0000_UUID
tfs-OST0001_UUID
tfs-OST0000_UUID
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
rename tfs to lustre
checking for existing Lustre data: found

   Read previous values:
Target:     tfs-MDT0000
Index:      0
Lustre FS:  tfs
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

cmd: tune2fs -f -L 'lustre-MDT0000' '/dev/mapper/mds1_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
checking for existing Lustre data: found

   Read previous values:
Target:     tfs-MDT0001
Index:      1
Lustre FS:  tfs
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

cmd: tune2fs -f -L 'lustre-MDT0001' '/dev/mapper/mds2_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
checking for existing Lustre data: found

   Read previous values:
Target:     tfs-OST0000
Index:      0
Lustre FS:  tfs
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

cmd: tune2fs -f -L 'lustre-OST0000' '/dev/mapper/ost1_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
checking for existing Lustre data: found

   Read previous values:
Target:     tfs-OST0001
Index:      1
Lustre FS:  tfs
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

cmd: tune2fs -f -L 'lustre-OST0001' '/dev/mapper/ost2_flakey' >/dev/null 2>&1
Writing CONFIGS/mountdata
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff88012dc1a800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff88012dc1a800.idle_timeout=debug
disable quota as required
PASS 103 (231s)

== conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 07:38:16 (1739536696)
mountfsopt: acl,user_xattr
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg238-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
Starting mds1: -o localrecov,noacl  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
Starting mds2: -o localrecov,noacl  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
setfacl: /mnt/lustre: Operation not supported
PASS 104a (68s)

== conf-sanity test 104b: Mount uses last flock argument ========================================================== 07:39:24 (1739536764)
mount lustre with opts flock,localflock on /mnt/lustre3.....
Starting client: oleg238-client.virtnet:  -o flock,localflock 192.168.202.138@tcp:/lustre /mnt/lustre3
192.168.202.138@tcp:/lustre on /mnt/lustre3 type lustre (rw,checksum,localflock,nouser_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
umount lustre on /mnt/lustre3.....
Stopping client oleg238-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock on /mnt/lustre3.....
Starting client: oleg238-client.virtnet:  -o localflock,flock 192.168.202.138@tcp:/lustre /mnt/lustre3
192.168.202.138@tcp:/lustre on /mnt/lustre3 type lustre (rw,checksum,flock,nouser_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
umount lustre on /mnt/lustre3.....
Stopping client oleg238-client.virtnet /mnt/lustre3 (opts:)
mount lustre with opts localflock,flock,noflock on /mnt/lustre3.....
Starting client: oleg238-client.virtnet:  -o localflock,flock,noflock 192.168.202.138@tcp:/lustre /mnt/lustre3
umount lustre on /mnt/lustre3.....
Stopping client oleg238-client.virtnet /mnt/lustre3 (opts:)
PASS 104b (3s)

== conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 07:39:27 (1739536767)
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
oleg238-server: rmmod: ERROR: Module lustre is in use
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
modules unloaded.
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
touch: cannot touch '/tmp/d105.conf-sanity/f105.conf-sanity': Read-only file system
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 105 (95s)

 SKIP: conf-sanity test_106 skipping SLOW test 106

== conf-sanity test 107: Unknown config param should not fail target mounting ========================================================== 07:41:03 (1739536863)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 107 (112s)

== conf-sanity test 108a: migrate from ldiskfs to ZFS ==== 07:42:55 (1739536975)

 SKIP: conf-sanity test_108a zfs only test
SKIP 108a (0s)

== conf-sanity test 108b: migrate from ZFS to ldiskfs ==== 07:42:55 (1739536975)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg238-server: 1+0 records in
oleg238-server: 1+0 records out
oleg238-server: 1048576 bytes (1.0 MB) copied, 0.00221337 s, 474 MB/s
oleg238-server: 1+0 records in
oleg238-server: 1+0 records out
oleg238-server: 1048576 bytes (1.0 MB) copied, 0.00219876 s, 477 MB/s
oleg238-server: 1+0 records in
oleg238-server: 1+0 records out
oleg238-server: 1048576 bytes (1.0 MB) copied, 0.00269819 s, 389 MB/s
oleg238-server: 1+0 records in
oleg238-server: 1+0 records out
oleg238-server: 1048576 bytes (1.0 MB) copied, 0.0029157 s, 360 MB/s

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters:

formatting backing filesystem ldiskfs on /dev/loop0
	target name   lustre-MDT0000
	kilobytes     200000
	options        -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDT0000  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/loop0 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x41
              (MDT update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp

formatting backing filesystem ldiskfs on /dev/loop0
	target name   lustre-MDT0001
	kilobytes     200000
	options        -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDT0001  -I 1024 -i 2560 -q -O uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/loop0 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x42
              (OST update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp

formatting backing filesystem ldiskfs on /dev/loop0
	target name   lustre-OST0000
	kilobytes     200000
	options        -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E resize="4290772992",lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E resize="4290772992",lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/loop0 200000k
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x42
              (OST update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp

formatting backing filesystem ldiskfs on /dev/loop0
	target name   lustre-OST0001
	kilobytes     200000
	options        -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E resize="4290772992",lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0001  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E resize="4290772992",lazy_journal_init,lazy_itable_init="0",packed_meta_blocks -F /dev/loop0 200000k
Writing CONFIGS/mountdata
changing server nid...
mounting mdt1 from backup...
mounting mdt2 from backup...
mounting ost1 from backup...
mounting ost2 from backup...
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Started LFSCK on the device lustre-OST0000: scrub
Started LFSCK on the device lustre-OST0001: scrub
mounting client...
check list
total 12
drwxr-xr-x 2 root root 4096 Jan 20  2018 d1
-rw-r--r-- 1 root root    0 Jan 20  2018 f0
-rw-r--r-- 1 root root 4067 Jan 20  2018 README
-rw-r--r-- 1 root root  331 Jan 20  2018 regression
check truncate && write
check create
check read && write && append
verify data
done.
cleanup...
PASS 108b (72s)

== conf-sanity test 109a: test lctl clear_conf fsname ==== 07:44:07 (1739537047)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Setting lustre-MDT0000.mdd.atime_diff from 60 to 62
Waiting 90s for '62'
Updated after 8s: want '62' got '62'
Setting lustre-MDT0000.mdd.atime_diff from 62 to 63
Waiting 90s for '63'
Updated after 5s: want '63' got '63'
Setting lustre.llite.max_read_ahead_mb from 256 to 32
Waiting 90s for '32'
Updated after 4s: want '32' got '32'
Setting lustre.llite.max_read_ahead_mb from 32 to 64
Waiting 90s for '64'
Waiting 80s for '64'
Updated after 11s: want '64' got '64'
oleg238-server: Pool lustre.pool1 created
Waiting 90s for ''
oleg238-server: OST lustre-OST0000_UUID added to pool lustre.pool1
oleg238-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg238-server: OST lustre-OST0000_UUID added to pool lustre.pool1
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
start mds service on oleg238-server
Starting mds1: -o localrecov -o nosvc  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
Start /dev/mapper/mds1_flakey without service
Started lustre-MDT0000
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Destroy the created pools: pool1
lustre.pool1
oleg238-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg238-server: Pool lustre.pool1 destroyed
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 109a (185s)

== conf-sanity test 109b: test lctl clear_conf one config ========================================================== 07:47:12 (1739537232)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Setting lustre-MDT0000.mdd.atime_diff from 60 to 62
Waiting 90s for '62'
Updated after 2s: want '62' got '62'
Setting lustre-MDT0000.mdd.atime_diff from 62 to 63
Waiting 90s for '63'
Updated after 5s: want '63' got '63'
Setting lustre.llite.max_read_ahead_mb from 256 to 32
Waiting 90s for '32'
Updated after 9s: want '32' got '32'
Setting lustre.llite.max_read_ahead_mb from 32 to 64
Waiting 90s for '64'
Updated after 6s: want '64' got '64'
oleg238-server: Pool lustre.pool1 created
Waiting 90s for ''
oleg238-server: OST lustre-OST0000_UUID added to pool lustre.pool1
oleg238-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg238-server: OST lustre-OST0000_UUID added to pool lustre.pool1
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
start mds service on oleg238-server
Starting mds1: -o localrecov -o nosvc  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
Start /dev/mapper/mds1_flakey without service
Started lustre-MDT0000
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Destroy the created pools: pool1
lustre.pool1
oleg238-server: OST lustre-OST0000_UUID removed from pool lustre.pool1
oleg238-server: Pool lustre.pool1 destroyed
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 109b (171s)

== conf-sanity test 110: Adding large_dir with 3-level htree ========================================================== 07:50:03 (1739537403)
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format ost1: /dev/mapper/ost1_flakey
MDT params: --mgs --fsname=lustre --mdt 			--index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity --backfstype=ldiskfs --device-size=200000  					 			--mkfsoptions=\"-b 1024 -i 65536 -O large_dir -b 4096 -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey 
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

device size = 2500MB
formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey
	target name   lustre:MDT0000
	kilobytes     200000
	options       -b 1024 -i 65536  -b 4096  -I 1024 -q -O large_dir,uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 1024 -L lustre:MDT0000 -b 1024 -i 65536  -b 4096  -I 1024 -q -O large_dir,uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 200000k
Writing CONFIGS/mountdata
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=131072
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
mount lustre  on /mnt/lustre2.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre2
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre2
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
mkdir: created directory '/mnt/lustre/d110.conf-sanity'
creating 60000 in total
creating 40000 hard links to /mnt/lustre/d110.conf-sanity/f110.conf-sanity-0
creating 20000 hard links to /mnt/lustre2/d110.conf-sanity/f110.conf-sanity-1
waiting for PIDs 4485 4504 to complete
 - link 6828 (time 1739537448.55 total 10.00 last 682.78)
 - link 10000 (time 1739537453.15 total 14.60 last 689.88)
 - link 16767 (time 1739537463.15 total 24.60 last 676.62)
total: 20000 link in 29.60 seconds: 675.79 ops/second
 - link 6828 (time 1739537448.55 total 10.00 last 682.77)
 - link 10000 (time 1739537453.15 total 14.60 last 689.77)
 - link 16768 (time 1739537463.15 total 24.60 last 676.78)
 - link 20000 (time 1739537468.14 total 29.59 last 647.57)
 - link 22200 (time 1739537478.14 total 39.59 last 219.99)
 - link 24300 (time 1739537488.14 total 49.59 last 209.95)
 - link 26431 (time 1739537498.14 total 59.59 last 213.08)
 - link 28588 (time 1739537508.14 total 69.60 last 215.67)
 - link 30000 (time 1739537514.73 total 76.18 last 214.47)
 - link 32204 (time 1739537524.73 total 86.18 last 220.38)
 - link 34441 (time 1739537534.73 total 96.19 last 223.58)
 - link 36617 (time 1739537544.74 total 106.19 last 217.51)
 - link 38797 (time 1739537554.74 total 116.19 last 217.90)
total: 40000 link in 121.84 seconds: 328.31 ops/second
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre2 opts:
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 2)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pasoleg238-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (26574848, 266) != expected (26583040, 266)
oleg238-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (26574848, 266) != expected (26583040, 266)
oleg238-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (26574848, 266) != expected (26583040, 266)
s1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 159 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 160 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 161 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 162 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 163 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 164 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 165 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 167 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 168 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 169 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 171 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 172 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 173 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 178 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 179 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 1596 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] Pass 1: Memory used: 416k/288k (270k/147k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 168.44MB/s
[Thread 0] Scanned group range [0, 2), inodes 281
Pass 2: Checking directory structure
Pass 2: Memory used: 416k/0k (82k/335k), time:  0.16/ 0.13/ 0.03
Pass 2: I/O read: 25MB, write: 0MB, rate: 155.49MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 416k/0k (83k/334k), time:  0.17/ 0.14/ 0.03
Pass 3: Memory used: 416k/0k (81k/336k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 416k/0k (68k/349k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 1MB, write: 0MB, rate: 7874.02MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 416k/0k (67k/350k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 833.33MB/s
Update quota info for quota type 0? no

Update quota info for quota type 1? no

Update quota info for quota type 2? no


lustre-MDT0000: ********** WARNING: Filesystem still has errors **********


         276 inodes used (8.85%, out of 3120)
           3 non-contiguous files (1.1%)
           1 non-contiguous directory (0.4%)
             # of inodes with ind/dind/tind blocks: 1/1/0
       11410 blocks used (22.82%, out of 50000)
           0 bad blocks
           1 large file

         148 regular files
         118 directories
           0 character device files
           0 block device files
           0 fifos
       60000 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
       60264 files
Memory usepdsh@oleg238-client: oleg238-server: ssh exited with exit code 4
d: 416k/0k (66k/351k), time:  0.17/ 0.14/ 0.03
I/O read: 25MB, write: 0MB, rate: 144.97MB/s
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 110 (211s)

 SKIP: conf-sanity test_111 skipping SLOW test 111

== conf-sanity test 112a: mount OST with no_create option ========================================================== 07:53:34 (1739537614)
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg238-server
Starting ost2: -o localrecov,no_create  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012a5e7800.ost_server_uuid 50
osc.lustre-OST0000-osc-ffff88012a5e7800.ost_server_uuid in FULL state after 0 sec
oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a5e7800.ost_server_uuid 50
osc.lustre-OST0001-osc-ffff88012a5e7800.ost_server_uuid in FULL state after 0 sec
lfs setstripe: setstripe error for '/mnt/lustre/f112a.conf-sanity.1': Invalid argument
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        95248        1644       84948   2% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        95248        1508       85084   2% /mnt/lustre[MDT:1]
lustre-OST0000_UUID       142216        1524      126692   2% /mnt/lustre[OST:0]
lustre-OST0001_UUID       142216        1116      127100   1% /mnt/lustre[OST:1] N

filesystem_summary:       284432        2640      253792   2% /mnt/lustre

obdfilter.lustre-OST0001.no_create=0
stop ost2 service on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 112a (60s)

== conf-sanity test 112b: mount MDT with no_create option ========================================================== 07:54:34 (1739537674)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start mds service on oleg238-server
Starting mds2: -o localrecov -o no_create  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg238-server
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0001
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
oleg238-server: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID        95248        1668       84924   2% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID        95248        1532       85060   2% /mnt/lustre[MDT:1] N
lustre-OST0000_UUID       142216        1524      126692   2% /mnt/lustre[OST:0]
lustre-OST0001_UUID       142216        1524      126692   2% /mnt/lustre[OST:1]

filesystem_summary:       284432        3048      253384   2% /mnt/lustre

    100 0
mdt.lustre-MDT0001.no_create=0
    100 1
stop ost2 service on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 112b (127s)

== conf-sanity test 113: Shadow mountpoint correctly report ro/rw for mounts ========================================================== 07:56:41 (1739537801)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff88012a5e6800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff88012a5e6800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 6s: want 'procname_uid' got 'procname_uid'
disable quota as required
/dev/mapper/mds1_flakey on /mnt/lustre-mds1 type lustre (rw,svname=lustre-MDT0000,mgs,osd=osd-ldiskfs,user_xattr,errors=remount-ro)
/dev/mapper/mds2_flakey on /mnt/lustre-mds2 type lustre (rw,svname=lustre-MDT0001,mgsnode=192.168.202.138@tcp,osd=osd-ldiskfs)
/dev/mapper/ost1_flakey on /mnt/lustre-ost1 type lustre (rw,svname=lustre-OST0000,mgsnode=192.168.202.138@tcp,osd=osd-ldiskfs)
/dev/mapper/ost2_flakey on /mnt/lustre-ost2 type lustre (rw,svname=lustre-OST0001,mgsnode=192.168.202.138@tcp,osd=osd-ldiskfs)
/dev/mapper/ost1_flakey on /mnt/lustre-ost1 type lustre (rw,svname=lustre-OST0000,mgsnode=192.168.202.138@tcp,osd=osd-ldiskfs)
/dev/mapper/ost2_flakey on /mnt/lustre-ost2 type lustre (rw,svname=lustre-OST0001,mgsnode=192.168.202.138@tcp,osd=osd-ldiskfs)
Shadow Mountpoint correctly reports rw for ldiskfs
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b58ce800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b58ce800.idle_timeout=debug
disable quota as required
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
PASS 113 (139s)

 SKIP: conf-sanity test_114 skipping SLOW test 114

 SKIP: conf-sanity test_115 skipping excluded test 115

== conf-sanity test 116: big size MDT support ============ 07:59:01 (1739537941)
/usr/sbin/mkfs.xfs
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
meta-data=/tmp/f116.conf-sanity-mdt0 isize=512    agcount=4, agsize=67108864 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=268435456, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=131072, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: not found
formatting backing filesystem ldiskfs on /dev/loop1
	target name   lustre:MDT0000
	kilobytes     18253611008
	options       -i 16777216 -b 4096  -J size=4096 -I 1024 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,64bit,^resize_inode,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,lazy_itable_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -i 16777216 -b 4096  -J size=4096 -I 1024 -q -O uninit_bg,extents,dirdata,dir_nlink,quota,project,huge_file,64bit,^resize_inode,ea_inode,large_dir,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,lazy_itable_init,packed_meta_blocks -F /dev/loop1 18253611008k
Writing CONFIGS/mountdata
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 116 (58s)

== conf-sanity test 117: lctl get_param return errors properly ========================================================== 07:59:59 (1739537999)
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
ost.OSS.ost_io.nrs_policies=fifo
oleg238-server: error: read_param: '/sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule': No such device
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 19
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 117 (35s)

== conf-sanity test 119: writeconf on slave mdt shouldn't duplicate mdc/osp and crash ========================================================== 08:00:34 (1739538034)
oleg238-server: error: get_param: param_path 'debug': No such file or directory
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
oleg238-server: error: set_param: param_path 'debug': No such file or directory
oleg238-server: error: set_param: setting 'debug'='+config': No such file or directory
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
oleg238-server: opening /dev/lnet failed: No such file or directory
oleg238-server: hint: the kernel modules may not be loaded
oleg238-server: IOC_LIBCFS_CLEAR_DEBUG failed: No such file or directory
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
debug_mb=84
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 300s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 270s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 260s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 250s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 240s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 230s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 220s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 210s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 200s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 190s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 180s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 170s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 150s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 140s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 130s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 120s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 110s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 100s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 90s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 80s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 70s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 60s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 50s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 20s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 0s for '1'
Update not seen after 300s: want '1' got '0'
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
debug_mb=84
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 300s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 290s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 280s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 250s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 240s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 230s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 220s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 130s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 120s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 110s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 100s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 90s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 80s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 70s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 60s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 50s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 40s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 10s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 0s for '1'
Update not seen after 300s: want '1' got '0'
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
debug_mb=84
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 300s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 280s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 270s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 260s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 250s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 240s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 230s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 220s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 210s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 200s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 190s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 180s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 170s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 130s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 120s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 110s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 90s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 80s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 70s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 60s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 50s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 40s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 30s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 20s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 10s for '1'
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Waiting 0s for '1'
Update not seen after 300s: want '1' got '0'
debug_mb=21
debug_mb=21
debug=-config
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg238-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
PASS 119 (1006s)

== conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 08:17:20 (1739539040)
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 3)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 82 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 84 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 121 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 123 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 125 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 126 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 127 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 159 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 160 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 161 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 26718 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 26719 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 26720 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 26722 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 26723 badness 0 to 2 for 10084
[Thread 0] group 2 finished
[Thread 0] e2fsck_pass1_run:2564: increase inode 53337 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53378 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53379 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 53380 badness 0 to 2 for 10084
[Thread 0] group 3 finished
[Thread 0] Pass 1: Memory used: 268k/0k (140k/129k), time:  0.00/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 293.86MB/s
[Thread 0] Scanned group range [0, 3), inodes 280
Pass 2: Checking directory structure
Pass 2: Memory used: 268k/0k (97k/172k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 284.33MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 268k/0k (98k/171k), time:  0.01/ 0.01/ 0.00
Pass 3: Memory used: 268k/0k (96k/173k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 268k/0k (67k/202k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 268k/0k (67k/202k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 310.37MB/s

         279 inodes used (0.35%, out of 79992)
           5 non-contiguous files (1.8%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
       24583 blocks used (49.17%, out of 50000)
           0 bad blocks
           1 large file

         149 regular files
         119 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           1 symbolic link (1 fast symbolic link)
           0 sockets
------------
         269 files
Memory used: 268k/0k (66k/203k), time:  0.02/ 0.01/ 0.00
I/O read: 1MB, write: 0MB, rate: 49.39MB/s
PASS 120 (51s)

== conf-sanity test 121: failover MGS ==================== 08:18:11 (1739539091)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Failing mgs on oleg238-server
Stopping /mnt/lustre-mds1 (opts:) on oleg238-server
08:18:26 (1739539106) shut down
facet: mgs facet_host: oleg238-server facet_failover_host: oleg238-server
Failover mgs to oleg238-server
mount facets: mgs
Starting mgs:   /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
08:18:39 (1739539119) targets are mounted
08:18:39 (1739539119) facet_failover done
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 95
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 95
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Failing mgs on oleg238-server
Stopping /mnt/lustre-mds1 (opts:) on oleg238-server
08:19:00 (1739539140) shut down
facet: mgs facet_host: oleg238-server facet_failover_host: oleg238-server
Failover mgs to oleg238-server
mount facets: mgs
Starting mgs:   /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
08:19:14 (1739539154) targets are mounted
08:19:14 (1739539154) facet_failover done
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 95
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 95
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 121 (76s)

== conf-sanity test 122a: Check OST sequence update ====== 08:19:27 (1739539167)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
fail_loc=0x00001e0
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
fail_loc=0
total: 1000 open/close in 2.70 seconds: 370.23 ops/second
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 122a (60s)

== conf-sanity test 123aa: llog_print works with FIDs and simple names ========================================================== 08:20:27 (1739539227)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
  1 UP mgs MGS MGS 7
- { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID }
- { index: 3, event: setup, device: lustre-clilov, UUID:  }
- { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID }
- { index: 7, event: setup, device: lustre-clilmv, UUID:  }
- { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv }
- { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID }
- { index: 3, event: setup, device: lustre-clilov, UUID:  }
- { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID }
- { index: 7, event: setup, device: lustre-clilmv, UUID:  }
- { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv }
- { index: 13, event: add_uuid, nid: 192.168.202.138@tcp(0x20000c0a8ca8a), node: 192.168.202.138@tcp }
- { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID }
- { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.202.138@tcp }
- { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID }
- { index: 22, event: add_uuid, nid: 192.168.202.138@tcp(0x20000c0a8ca8a), node: 192.168.202.138@tcp }
- { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID }
- { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.202.138@tcp }
- { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID }
- { index: 31, event: add_uuid, nid: 192.168.202.138@tcp(0x20000c0a8ca8a), node: 192.168.202.138@tcp }
- { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }
- { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.138@tcp }
- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }
- { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 }
PASS 123aa (31s)

== conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:20:58 (1739539258)
PASS 123ab (3s)

== conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:21:01 (1739539261)
PASS 123ac (2s)

== conf-sanity test 123ad: llog_print shows all records == 08:21:03 (1739539263)
PASS 123ad (3s)

== conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:21:07 (1739539267)
- { index: 11, event: set_param, device: general, parameter: osc.*.max_dirty_mb, value: 467 }
- { index: 46, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_dirty_mb=467 }
- { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID }
- { index: 3, event: setup, device: lustre-clilov, UUID:  }
- { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID }
- { index: 7, event: setup, device: lustre-clilmv, UUID:  }
- { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv }
- { index: 13, event: add_uuid, nid: 192.168.202.138@tcp(0x20000c0a8ca8a), node: 192.168.202.138@tcp }
- { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID }
- { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.202.138@tcp }
- { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID }
- { index: 22, event: add_uuid, nid: 192.168.202.138@tcp(0x20000c0a8ca8a), node: 192.168.202.138@tcp }
- { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID }
- { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.202.138@tcp }
- { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID }
- { index: 31, event: add_uuid, nid: 192.168.202.138@tcp(0x20000c0a8ca8a), node: 192.168.202.138@tcp }
- { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }
- { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.138@tcp }
- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }
- { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 }
- { index: 43, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_pages_per_rpc=1024 }
- { index: 46, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_dirty_mb=467 }
- { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID }
- { index: 3, event: setup, device: lustre-clilov, UUID:  }
- { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID }
- { index: 7, event: setup, device: lustre-clilmv, UUID:  }
- { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv }
- { index: 13, event: add_uuid, nid: 192.168.202.138@tcp(0x20000c0a8ca8a), node: 192.168.202.138@tcp }
- { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID }
- { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.202.138@tcp }
- { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID }
- { index: 22, event: add_uuid, nid: 192.168.202.138@tcp(0x20000c0a8ca8a), node: 192.168.202.138@tcp }
- { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID }
- { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.202.138@tcp }
- { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID }
- { index: 31, event: add_uuid, nid: 192.168.202.138@tcp(0x20000c0a8ca8a), node: 192.168.202.138@tcp }
- { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }
- { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.138@tcp }
- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }
- { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 }
- { index: 43, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_pages_per_rpc=1024 }
PASS 123ae (8s)

== conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:21:14 (1739539274)
lctl --device MGS llog_catlist ...
orig_clist: lustre-MDT0000 lustre-OST0000 lustre-client lustre-MDT0001
fail_loc=0x131b
fail_val=2
new_clist: lustre-OST0000 lustre-client lustre-MDT0001
fail_loc=0
done
lctl --device lustre-MDT0000 llog_catlist ...
orig_clist: [0x1:0x2:0x0]
fail_loc=0x131b
fail_val=2
new_clist: 
fail_loc=0
done
fail_loc=0
PASS 123af (4s)

== conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:21:18 (1739539278)
PASS 123ag (4s)

== conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:21:22 (1739539282)
del_ost: dry run for target lustre-OST0000
config_log: lustre-MDT0000
[DRY RUN] cancel catalog 'lustre-MDT0000:41':"- { index: 41, event: conf_param, device: lustre-OST0000-osc-MDT0000, parameter: osc.max_dirty_mb=467 }"
[DRY RUN] cancel catalog 'lustre-MDT0000:29':"- { index: 29, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-MDT0000:28':"- { index: 28, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.202.138@tcp }"
[DRY RUN] cancel catalog 'lustre-MDT0000:27':"- { index: 27, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-client
[DRY RUN] cancel catalog 'lustre-client:34':"- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-client:33':"- { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.138@tcp }"
[DRY RUN] cancel catalog 'lustre-client:32':"- { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-MDT0001
[DRY RUN] cancel catalog 'lustre-MDT0001:44':"- { index: 44, event: conf_param, device: lustre-OST0000-osc-MDT0001, parameter: osc.max_dirty_mb=467 }"
[DRY RUN] cancel catalog 'lustre-MDT0001:32':"- { index: 32, event: add_osc, device: lustre-MDT0001-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }"
[DRY RUN] cancel catalog 'lustre-MDT0001:31':"- { index: 31, event: setup, device: lustre-OST0000-osc-MDT0001, UUID: lustre-OST0000_UUID, node: 192.168.202.138@tcp }"
[DRY RUN] cancel catalog 'lustre-MDT0001:30':"- { index: 30, event: attach, device: lustre-OST0000-osc-MDT0001, type: osc, UUID: lustre-MDT0001-mdtlov_UUID }"
del_ost: no catalog entry deleted
config_log: lustre-MDT0000
cancel catalog lustre-MDT0000 log_idx 41: done
cancel catalog lustre-MDT0000 log_idx 29: done
cancel catalog lustre-MDT0000 log_idx 28: done
cancel catalog lustre-MDT0000 log_idx 27: done
del_ost: cancelled 4 catalog entries
config_log: lustre-client
cancel catalog lustre-client log_idx 34: done
cancel catalog lustre-client log_idx 33: done
cancel catalog lustre-client log_idx 32: done
del_ost: cancelled 3 catalog entries
config_log: lustre-MDT0001
cancel catalog lustre-MDT0001 log_idx 44: done
cancel catalog lustre-MDT0001 log_idx 32: done
cancel catalog lustre-MDT0001 log_idx 31: done
cancel catalog lustre-MDT0001 log_idx 30: done
del_ost: cancelled 4 catalog entries
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
fail_loc=0
PASS 123ah (80s)

== conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:22:42 (1739539362)
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
oleg238-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
- { index: 394, event: set_param, device: general, parameter: timeout, value: 129 }
cleanup test 123ai
timeout=20
timeout=20
PASS 123ai (54s)

== conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:23:36 (1739539416)
oleg238-server: rm: cannot remove '/tmp/f123F.conf-sanity.yaml': No such file or directory
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Unmounting FS
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Writeconf
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x162
              (OST first_time update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

Remounting
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Setting configuration parameters
oleg238-server: warning: 'lctl set_param -F' is deprecated, use 'lctl apply_yaml' instead
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 123F (84s)

== conf-sanity test 123G: clear and reset all parameters using apply_yaml ========================================================== 08:25:00 (1739539500)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
oleg238-server: rm: cannot remove '/tmp/f123G.conf-sanity.yaml': No such file or directory
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Unmounting FS
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Writeconf
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x105
              (MDT MGS writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x102
              (OST writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x142
              (OST update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x142
              (OST update writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

Remounting
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Setting configuration parameters
conf_param: lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
conf_param: lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
set_param: jobid_var=TESTNAME
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 123G (103s)

== conf-sanity test 123H: check concurent accesses with 'lctl llog_print ========================================================== 08:26:43 (1739539603)
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
     20 702
PASS 123H (59s)

== conf-sanity test 124: check failover after replace_nids ========================================================== 08:27:42 (1739539662)

 SKIP: conf-sanity test_124 needs MDT failover setup
SKIP 124 (1s)

== conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:27:43 (1739539663)
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
oleg238-server: oleg238-server.virtnet: executing load_module ../libcfs/libcfs/libcfs
fail_loc=0x60d
oleg238-server: oleg238-server.virtnet: executing load_modules
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
clearing fail_loc on mds1
fail_loc=0
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
PASS 126 (50s)

== conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:28:33 (1739539713)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Stopping clients: /mnt/lustre  (opts:)
pdsh@oleg238-client: no remote hosts specified
osc.lustre-OST0000-osc-MDT0000.active updated after 0 sec (got 1)
osc.lustre-OST0000-osc-MDT0001.active updated after 0 sec (got 1)
Waiting for MDT destroys to complete
oleg238-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg238-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
dd: error writing '/mnt/lustre/f127.conf-sanity': No space left on device
124+0 records in
123+0 records out
128974848 bytes (129 MB) copied, 2.46636 s, 52.3 MB/s
123+0 records in
123+0 records out
128974848 bytes (129 MB) copied, 2.62015 s, 49.2 MB/s
PASS 127 (54s)

== conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:29:27 (1739539767)

 SKIP: conf-sanity test_128 need separate mgs device
SKIP 128 (1s)

== conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:29:28 (1739539768)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Format ost1: /dev/mapper/ost1_flakey
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg238-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Address already in use
oleg238-server: The target service's index is already in use. (/dev/mapper/ost1_flakey)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 98
oleg238-server: error: set_param: param_path 'seq/cli-lustre:OST0000-super/width': No such file or directory
oleg238-server: error: set_param: setting 'seq/cli-lustre:OST0000-super/width'='65536': No such file or directory
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 98
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg238-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Address already in use
oleg238-server: The target service's index is already in use. (/dev/mapper/ost1_flakey)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 98
oleg238-server: error: set_param: param_path 'seq/cli-lustre:OST0000-super/width': No such file or directory
oleg238-server: error: set_param: setting 'seq/cli-lustre:OST0000-super/width'='65536': No such file or directory
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 98
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x22
              (OST first_time )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20


   Permanent disk data:
Target:     lustre=OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x122
              (OST first_time writeconf )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

Writing CONFIGS/mountdata
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Stopping /mnt/lustre-ost1 (opts:) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 129 (53s)

== conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:30:21 (1739539821)
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b58e0800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b58e0800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 3s: want 'procname_uid' got 'procname_uid'
disable quota as required
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x1
              (MDT )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre=MDT0001
Index:      1
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x101
              (MDT writeconf )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

Writing CONFIGS/mountdata
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
PASS 130 (45s)

== conf-sanity test 131: MDT backup restore with project ID ========================================================== 08:31:06 (1739539866)
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg238-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg238-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
mount.lustre: according to /etc/mtab 192.168.202.138@tcp:/lustre is already mounted on /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b58e0800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b58e0800.idle_timeout=debug
disable quota as required
striped dir -i1 -c2 -H crush2 /mnt/lustre/d131.conf-sanity
total: 512 open/close in 1.39 seconds: 369.06 ops/second
striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d131.conf-sanity.inherit
total: 128 open/close in 0.29 seconds: 440.78 ops/second
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff88012a5dc800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff88012a5dc800.idle_timeout=debug
disable quota as required
PASS 131 (145s)

== conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:33:31 (1739540011)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg238-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x5
              (MDT MGS )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity


   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mdt.hsm_control=enabled

Writing CONFIGS/mountdata
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
PASS 132 (82s)

== conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:34:53 (1739540093)

 SKIP: conf-sanity test_133 needs >= 4 OSTs
SKIP 133 (1s)

== conf-sanity test 134: check_iam works without faults == 08:34:54 (1739540094)
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800aa92e800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800aa92e800.idle_timeout=debug
disable quota as required
striped dir -i0 -c2 -H crush2 /mnt/lustre/d134.conf-sanity
Started LFSCK on the device lustre-MDT0000: scrub layout namespace
	idle_blocks: 0
Root entries: count 2, limit 203
000: [0:0x0:0x0]	ptr: 1
Block 1,FIX leaf, Leaf block, count 1, limit 240
000: [0:0x0:0x0]	0x0 ()
count 1
NO ERRORS
dd if=/dev/urandom of=/tmp/d134.conf-sanity/lfsck_namespace_13 bs=2 conv=notrunc count=1 seek=3
Bad magic 16229847208368517613
Filesize 8192, blocks count 2
Root format: Root node is insane
FINISHED WITH ERRORS
255
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 255
debugfs 1.46.2.wc5 (26-Mar-2022)
/dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Record type to print: lfsck_namespace
Filesize 8192, blocks count 2
Root format: LFIX
	keysize: 16
	recsize: 1
	ptrsize: 4
	indirect_levels: 0
	idle_blocks: 0
Root entries: count 2, limit 203
000: [0:0x0:0x0]	ptr: 1
Block 1,FIX leaf, Leaf block, count 1, limit 240
000: [0:0x0:0x0]	0x0 ()
count 1
NO ERRORS
dd if=/dev/urandom of=/tmp/d134.conf-sanity/lfsck_namespace_14 bs=2 conv=notrunc count=1 seek=0
Bad magic 51232100011
Filesize 8192, blocks count 2
Root format: Root node is insane
FINISHED WITH ERRORS
255
debugfs 1.46.2.wc5 (26-Mar-2022)
/dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Record type to print: lfsck_namespace
Filesize 8192, blocks count 2
Root format: LFIX
	keysize: 16
	recsize: 1
	ptrsize: 4
	indirect_levels: 0
	idle_blocks: 0
Root entries: count 2, limit 203
000: [0:0x0:0x0]	ptr: 1
Block 1,FIX leaf, Leaf block, count 1, limit 240
000: [0:0x0:0x0]	0x0 ()
count 1
NO ERRORS
dd if=/dev/urandom of=/tmp/d134.conf-sanity/lfsck_namespace_15 bs=2 conv=notrunc count=1 seek=28
Filesize 8192, blocks count 2
Root format: LFIX
	keysize: 16
	recsize: 1
	ptrsize: 4
	indirect_levels: 0
	idle_blocks: 0
Root entries: count 2, limit 203
000: [0:0x0:0x0]	ptr: 1
Block 1,FIX leaf, Leaf block, count 1, limit 240
count 1
NO ERRORS
0
debugfs 1.46.2.wc5 (26-Mar-2022)
/dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Record type to print: nodemap
Filesize 8192, blocks count 2
Root format: LFIX
	keysize: 8
	recsize: 32
	ptrsize: 4
	indirect_levels: 0
	idle_blocks: 0
Root entries: count 2, limit 340
000: { id: 0x0, type: empty(0) }	ptr: 1
Block 1,FIX leaf, Leaf block, count 5, limit 102
000: { id: 0x0, type: empty(0) }	{}
001: { id: 0x0, type: cluster(1), subtype: cluster }	{ name: default, flag: 0xb8, flag2: 0x0, squash_uid: 65534, squash_gid: 65534, squash_projid: 65534}
002: { id: 0x0, type: global(15) }	{ is_active: 0 }
003: { id: 0x1, type: cluster(1), subtype: cluster }	{ name: test_134, flag: 0xb8, flag2: 0x0, squash_uid: 65534, squash_gid: 65534, squash_projid: 65534}
004: { id: 0x1, type: range(2) }	{ start_nid: 121.23.2.100@tcp, end_nid: 121.23.2.120@tcp }
count 5
NO ERRORS
dd if=/dev/urandom of=/tmp/d134.conf-sanity/nodemap bs=2 conv=notrunc count=1 seek=33
Filesize 8192, blocks count 2
Root format: LFIX
	keysize: 8
	recsize: 32
	ptrsize: 4
	indirect_levels: 0
	idle_blocks: 0
Root entries: count 2, limit 340
000: { id: 0x0, type: empty(0) }	ptr: 1
Block 1,FIX leaf, Leaf block, count 5, limit 102
count 5
NO ERRORS
0
PASS 134 (210s)

== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:38:24 (1739540304)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg238-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
oleg238-client: fail_loc=0x1312
oleg238-client: fail_val=5
oleg238-server: fail_loc=0x1312
oleg238-server: fail_val=5
striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
total: 4500 open/close in 9.50 seconds: 473.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #12998
total: 4500 /unlink in 4.78 seconds: 942.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
total: 4500 open/close in 9.60 seconds: 468.66 ops/second
total: 4500 /unlink in 4.93 seconds: 912.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 4485 (time 1739540416.66 total 10.00 last 448.45)
total: 4500 open/close in 10.03 seconds: 448.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
total: 4500 /unlink in 4.87 seconds: 923.97 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11028: 26739 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
PASS 135 (146s)

== conf-sanity test 136: don't panic with bad obdecho setup ========================================================== 08:40:50 (1739540450)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
oleg238-server: oleg238-server.virtnet: executing load_module obdecho/obdecho
oleg238-server: error: setup: Invalid argument
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 22
oleg238-server: error: test_mkdir: No such file or directory
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 136 (113s)

== conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:42:43 (1739540563)

 SKIP: conf-sanity test_137 needs >= 3 MDTs
SKIP 137 (1s)

== conf-sanity test 140: remove_updatelog script actions ========================================================== 08:42:44 (1739540564)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
striped dir -i0 -c2 -H crush2 /mnt/lustre/d140.conf-sanity
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Dry run was requested, no changes will be applied
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 0 1
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
Processing MDT1 llog catalog [0x240000400:0x1:0x0] ...
remove_updatelog: /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] is too small.
> /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0]
Dry run was requested, no changes will be applied
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 1 0
Processing MDT1 llog catalog [0x240000400:0x1:0x0] ...
remove_updatelog: /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0] is too small.
> /mnt/lustre-mds2/update_log_dir/[0x240000400:0x1:0x0]
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
Scan update_log at '/mnt/lustre-mds2':
Selected MDTS: 0
Processing MDT0 llog catalog [0x240000401:0x1:0x0] ...
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x2:0x0]
rm -f /mnt/lustre-mds2/update_log_dir/[0x240000bd0:0x3:0x0]
> /mnt/lustre-mds2/update_log_dir/[0x240000401:0x1:0x0]
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 140 (226s)

== conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:46:30 (1739540790)
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
llite.lustre-ffff8800a7a2e000.max_cached_mb=100%
llite.lustre-ffff8800a7a2e000.max_cached_mb=
users: 5
max_cached_mb: 3730
used_mb: 0
unused_mb: 3730
unevict_mb: 0
reclaim_count: 0
max_read_ahead_mb: 256
used_read_ahead_mb: 0
total ram mb: 3730
llite.lustre-ffff8800a7a2e000.max_cached_mb=50%
llite.lustre-ffff8800a7a2e000.max_cached_mb=
users: 5
max_cached_mb: 1865
used_mb: 0
unused_mb: 1865
unevict_mb: 0
reclaim_count: 0
max_read_ahead_mb: 256
used_read_ahead_mb: 0
error: set_param: setting /sys/kernel/debug/lustre/llite/lustre-ffff8800a7a2e000/max_cached_mb=105%: Numerical result out of range
error: set_param: setting 'llite/*/max_cached_mb'='105%': Numerical result out of range
llite.lustre-ffff8800a7a2e000.max_cached_mb=0%
llite.lustre-ffff8800a7a2e000.max_cached_mb=
users: 5
max_cached_mb: 64
used_mb: 0
unused_mb: 64
unevict_mb: 0
reclaim_count: 0
max_read_ahead_mb: 256
used_read_ahead_mb: 0
PASS 150 (20s)

== conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:46:50 (1739540810)
umount lustre on /mnt/lustre.....
Stopping client oleg238-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
Damage ost1 local config log
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
start ost1 service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg238-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Invalid argument
oleg238-server: This may have multiple causes.
oleg238-server: Are the mount options correct?
oleg238-server: Check the syslog for more info.
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 22
oleg238-server: error: set_param: param_path 'seq/cli-lustre-OST0000-super/width': No such file or directory
oleg238-server: error: set_param: setting 'seq/cli-lustre-OST0000-super/width'='65536': No such file or directory
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
Start of /dev/mapper/ost1_flakey on ost1 failed 22
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 151a (129s)

== conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:48:59 (1739540939)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
stop mds service on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
start mds service on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
fail_loc=0x131e
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 151b (92s)

== conf-sanity test 152: seq allocation error in OSP ===== 08:50:31 (1739541031)
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800a7a29000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800a7a29000.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 6s: want 'procname_uid' got 'procname_uid'
disable quota as required
striped dir -i1 -c1 -H fnv_1a_64 /mnt/lustre/d152.conf-sanity
ADD OST3

   Permanent disk data:
Target:     lustre:OST0003
Index:      3
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20

formatting backing filesystem ldiskfs on /dev/loop0
	target name   lustre:OST0003
	kilobytes     200000
	options       -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0003 -b 4096  -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/loop0 200000k
Writing CONFIGS/mountdata
fail_loc=0x80002109
fail_val=2
START OST3
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Starting ost3: -o localrecov  /dev/mapper/ost3_flakey /mnt/lustre-ost3
seq.cli-lustre-OST0003-super.width=65536
STOP OST3
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /tmp/lustre-ost3
Stopping /mnt/lustre-ost3 (opts:) on oleg238-server
30147
Started lustre-OST0003
fail_loc=0
START OST3 again
Starting ost3: -o localrecov  /dev/mapper/ost3_flakey /mnt/lustre-ost3
seq.cli-lustre-OST0003-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0003
/mnt/lustre/d152.conf-sanity/f152.conf-sanity-2
lmm_magic:         0x0BD10BD0
lmm_seq:           0x240000bd0
lmm_object_id:     0x3
lmm_fid:           [0x240000bd0:0x3:0x0]
lmm_stripe_count:  3
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	            35	         0x23	   0x280000400
	     1	             3	          0x3	   0x2c0000400
	     3	             2	          0x2	   0x300000401

Stopping /mnt/lustre-ost3 (opts:) on oleg238-server
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg238-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
PASS 152 (130s)

== conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:52:41 (1739541161)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
waiting for mount ...
          "192.168.202.138@tcp": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 5 }
          "192.168.252.112@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
          "10.252.252.113@tcp": { connects: 0, replied: 0, uptodate: discovering, sec_ago: never }
          "192.168.202.138@tcp": { connects: 0, replied: 0, uptodate: unknown, sec_ago: never }
setup single mount lustre success
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 153a (189s)

== conf-sanity test 153b: added IPv6 NID support ========= 08:55:50 (1739541350)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800a9c6d000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800a9c6d000.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 2s: want 'procname_uid' got 'procname_uid'
disable quota as required
PASS 153b (84s)

== conf-sanity test 153c: don't stuck on unreached NID === 08:57:14 (1739541434)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
waiting for mount ...
          "192.168.252.112@tcp": { connects: 11, replied: 0, uptodate: discovering, sec_ago: 0 }
          "192.168.252.113@tcp": { connects: 10, replied: 0, uptodate: discovering, sec_ago: 2 }
          "192.168.252.115@tcp": { connects: 10, replied: 0, uptodate: discovering, sec_ago: 1 }
connection attempts: 31
Waiting for mount to fail
mount.lustre: mount 192.168.252.112@tcp:192.168.252.113@tcp:192.168.252.115@tcp:/lustre at /mnt/lustre failed: Input/output error
Is the MGS running?
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
PASS 153c (344s)

== conf-sanity test 154: expand .. on rename after MDT backup restore ========================================================== 09:02:58 (1739541778)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff880136831800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff880136831800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 2s: want 'procname_uid' got 'procname_uid'
disable quota as required
striped dir -i0 -c1 -H all_char /mnt/lustre/d154.conf-sanity
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d154.conf-sanity.tgt
striped dir -i0 -c1 -H crush2 /mnt/lustre/d154.conf-sanity.dx_dir
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d154.conf-sanity.dirent
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d154.conf-sanity.empty
total: 128 open/close in 0.25 seconds: 504.61 ops/second
total: 2 open/close in 0.01 seconds: 134.60 ops/second
==target parent FID: fid:[0x200000402:0x1:0x0]==
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
==debugfs before backup==
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 53385   Type: directory    Mode:  0755   Flags: 0x1000
Generation: 4024968532    Version: 0x00000001:0000010c
User:     0   Group:     0   Project:     0   Size: 16384
File ACL: 0
Links: 2   Blockcount: 32
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x67af4d4c:00000000 -- Fri Feb 14 09:03:56 2025
 atime: 0x67af4d4b:00000000 -- Fri Feb 14 09:03:55 2025
 mtime: 0x67af4d4c:00000000 -- Fri Feb 14 09:03:56 2025
crtime: 0x67af4d4b:af3cd700 -- Fri Feb 14 09:03:55 2025
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x3:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  user.job (5) = "lfs.0"
  linkea: idx=0 parent=[0x200000402:0x1:0x0] name='d154.conf-sanity.dx_dir'
BLOCKS:
(0):24781, (1-3):24784-24786
TOTAL: 4

oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  53385   40755 (2)      0      0   16384 14-Feb-2025 09:03 .
  53383   40755 (18)      0      0    4096 14-Feb-2025 09:03 fid:[0x200000402:0x1:0x0] ..
    170  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0xb:0x0] f154.conf-sanity.longfilename5
    172  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0xd:0x0] f154.conf-sanity.longfilename7
    174  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0xf:0x0] f154.conf-sanity.longfilename9
    175  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x10:0x0] f154.conf-sanity.longfilename10
    176  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x11:0x0] f154.conf-sanity.longfilename11
    186  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x1b:0x0] f154.conf-sanity.longfilename21
    192  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x21:0x0] f154.conf-sanity.longfilename27
    193  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x22:0x0] f154.conf-sanity.longfilename28
    198  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x27:0x0] f154.conf-sanity.longfilename33
    201  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x2a:0x0] f154.conf-sanity.longfilename36
    204  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x2d:0x0] f154.conf-sanity.longfilename39
    206  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x2f:0x0] f154.conf-sanity.longfilename41
    209  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x32:0x0] f154.conf-sanity.longfilename44
    212  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x35:0x0] f154.conf-sanity.longfilename47
    214  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x37:0x0] f154.conf-sanity.longfilename49
    215  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x38:0x0] f154.conf-sanity.longfilename50
    218  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x3b:0x0] f154.conf-sanity.longfilename53
    219  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x3c:0x0] f154.conf-sanity.longfilename54
    222  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x3f:0x0] f154.conf-sanity.longfilename57
    226  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x43:0x0] f154.conf-sanity.longfilename61
    228  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x45:0x0] f154.conf-sanity.longfilename63
    231  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x48:0x0] f154.conf-sanity.longfilename66
    235  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x4c:0x0] f154.conf-sanity.longfilename70
    246  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x57:0x0] f154.conf-sanity.longfilename81
    247  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x58:0x0] f154.conf-sanity.longfilename82
    250  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x5b:0x0] f154.conf-sanity.longfilename85
    260  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x65:0x0] f154.conf-sanity.longfilename95
    264  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x69:0x0] f154.conf-sanity.longfilename99
    271  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x70:0x0] f154.conf-sanity.longfilename106
    272  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x71:0x0] f154.conf-sanity.longfilename107
    273  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x72:0x0] f154.conf-sanity.longfilename108
    278  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x77:0x0] f154.conf-sanity.longfilename113
    281  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x7a:0x0] f154.conf-sanity.longfilename116
    286  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x7f:0x0] f154.conf-sanity.longfilename121
    292  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x85:0x0] f154.conf-sanity.longfilename127
    207  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x30:0x0] f154.conf-sanity.longfilename42
    188  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x1d:0x0] f154.conf-sanity.longfilename23
    217  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x3a:0x0] f154.conf-sanity.longfilename52
    223  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x40:0x0] f154.conf-sanity.longfilename58
    216  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x39:0x0] f154.conf-sanity.longfilename51
    194  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x23:0x0] f154.conf-sanity.longfilename29
    179  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x14:0x0] f154.conf-sanity.longfilename14
    229  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x46:0x0] f154.conf-sanity.longfilename64
    180  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x15:0x0] f154.conf-sanity.longfilename15
    213  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x36:0x0] f154.conf-sanity.longfilename48
    181  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x16:0x0] f154.conf-sanity.longfilename16
    232  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x49:0x0] f154.conf-sanity.longfilename67
    197  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x26:0x0] f154.conf-sanity.longfilename32
    166  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x7:0x0] f154.conf-sanity.longfilename1
    182  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x17:0x0] f154.conf-sanity.longfilename17
    169  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0xa:0x0] f154.conf-sanity.longfilename4
    187  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x1c:0x0] f154.conf-sanity.longfilename22
    202  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x2b:0x0] f154.conf-sanity.longfilename37
    196  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x25:0x0] f154.conf-sanity.longfilename31
    184  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x19:0x0] f154.conf-sanity.longfilename19
    200  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x29:0x0] f154.conf-sanity.longfilename35
    224  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x41:0x0] f154.conf-sanity.longfilename59
    177  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x12:0x0] f154.conf-sanity.longfilename12
    165  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x6:0x0] f154.conf-sanity.longfilename0
    185  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x1a:0x0] f154.conf-sanity.longfilename20
    205  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x2e:0x0] f154.conf-sanity.longfilename40
    210  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x33:0x0] f154.conf-sanity.longfilename45
    183  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x18:0x0] f154.conf-sanity.longfilename18
    178  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x13:0x0] f154.conf-sanity.longfilename13
    191  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x20:0x0] f154.conf-sanity.longfilename26
    203  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x2c:0x0] f154.conf-sanity.longfilename38
    225  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x42:0x0] f154.conf-sanity.longfilename60
    199  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x28:0x0] f154.conf-sanity.longfilename34
    234  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x4b:0x0] f154.conf-sanity.longfilename69
    237  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x4e:0x0] f154.conf-sanity.longfilename72
    239  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x50:0x0] f154.conf-sanity.longfilename74
    240  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x51:0x0] f154.conf-sanity.longfilename75
    248  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x59:0x0] f154.conf-sanity.longfilename83
    249  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x5a:0x0] f154.conf-sanity.longfilename84
    251  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x5c:0x0] f154.conf-sanity.longfilename86
    252  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x5d:0x0] f154.conf-sanity.longfilename87
    253  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x5e:0x0] f154.conf-sanity.longfilename88
    258  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x63:0x0] f154.conf-sanity.longfilename93
    259  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x64:0x0] f154.conf-sanity.longfilename94
    261  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x66:0x0] f154.conf-sanity.longfilename96
    262  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x67:0x0] f154.conf-sanity.longfilename97
    266  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x6b:0x0] f154.conf-sanity.longfilename101
    267  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x6c:0x0] f154.conf-sanity.longfilename102
    269  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x6e:0x0] f154.conf-sanity.longfilename104
    274  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x73:0x0] f154.conf-sanity.longfilename109
    276  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x75:0x0] f154.conf-sanity.longfilename111
    277  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x76:0x0] f154.conf-sanity.longfilename112
    279  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x78:0x0] f154.conf-sanity.longfilename114
    280  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x79:0x0] f154.conf-sanity.longfilename115
    282  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x7b:0x0] f154.conf-sanity.longfilename117
    285  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x7e:0x0] f154.conf-sanity.longfilename120
    289  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x82:0x0] f154.conf-sanity.longfilename124
    290  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x83:0x0] f154.conf-sanity.longfilename125
    291  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x84:0x0] f154.conf-sanity.longfilename126
    268  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x6d:0x0] f154.conf-sanity.longfilename103
    244  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x55:0x0] f154.conf-sanity.longfilename79
    265  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x6a:0x0] f154.conf-sanity.longfilename100
    233  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x4a:0x0] f154.conf-sanity.longfilename68
    283  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x7c:0x0] f154.conf-sanity.longfilename118
    254  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x5f:0x0] f154.conf-sanity.longfilename89
    242  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x53:0x0] f154.conf-sanity.longfilename77
    245  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x56:0x0] f154.conf-sanity.longfilename80
    287  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x80:0x0] f154.conf-sanity.longfilename122
    275  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x74:0x0] f154.conf-sanity.longfilename110
    190  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x1f:0x0] f154.conf-sanity.longfilename25
    173  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0xe:0x0] f154.conf-sanity.longfilename8
    230  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x47:0x0] f154.conf-sanity.longfilename65
    284  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x7d:0x0] f154.conf-sanity.longfilename119
    171  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0xc:0x0] f154.conf-sanity.longfilename6
    220  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x3d:0x0] f154.conf-sanity.longfilename55
    168  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x9:0x0] f154.conf-sanity.longfilename3
    211  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x34:0x0] f154.conf-sanity.longfilename46
    257  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x62:0x0] f154.conf-sanity.longfilename92
    263  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x68:0x0] f154.conf-sanity.longfilename98
    167  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x8:0x0] f154.conf-sanity.longfilename2
    238  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x4f:0x0] f154.conf-sanity.longfilename73
    241  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x52:0x0] f154.conf-sanity.longfilename76
    236  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x4d:0x0] f154.conf-sanity.longfilename71
    288  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x81:0x0] f154.conf-sanity.longfilename123
    255  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x60:0x0] f154.conf-sanity.longfilename90
    270  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x6f:0x0] f154.conf-sanity.longfilename105
    227  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x44:0x0] f154.conf-sanity.longfilename62
    243  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x54:0x0] f154.conf-sanity.longfilename78
    208  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x31:0x0] f154.conf-sanity.longfilename43
    195  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x24:0x0] f154.conf-sanity.longfilename30
    221  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x3e:0x0] f154.conf-sanity.longfilename56
    189  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x1e:0x0] f154.conf-sanity.longfilename24
    256  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x61:0x0] f154.conf-sanity.longfilename91

oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 53386   Type: directory    Mode:  0755   Flags: 0x0
Generation: 4024968533    Version: 0x00000001:0000010d
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x67af4d4c:00000000 -- Fri Feb 14 09:03:56 2025
 atime: 0x67af4d4b:00000000 -- Fri Feb 14 09:03:55 2025
 mtime: 0x67af4d4c:00000000 -- Fri Feb 14 09:03:56 2025
crtime: 0x67af4d4b:b71b0000 -- Fri Feb 14 09:03:55 2025
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x4:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  user.job (5) = "lfs.0"
  linkea: idx=0 parent=[0x200000402:0x1:0x0] name='d154.conf-sanity.dirent'
BLOCKS:
(0):24782
TOTAL: 1

oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  53386   40755 (2)      0      0    4096 14-Feb-2025 09:03 .
  53383   40755 (18)      0      0    4096 14-Feb-2025 09:03 fid:[0x200000402:0x1:0x0] ..
    293  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x86:0x0] f154.conf-sanity.longfilename0
    294  100644 (17)      0      0       0 14-Feb-2025 09:03 fid:[0x200000402:0x87:0x0] f154.conf-sanity.longfilename1

oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 53387   Type: directory    Mode:  0755   Flags: 0x0
Generation: 4024968534    Version: 0x00000001:0000010e
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x67af4d4c:00000000 -- Fri Feb 14 09:03:56 2025
 atime: 0x67af4d4b:00000000 -- Fri Feb 14 09:03:55 2025
 mtime: 0x67af4d4b:00000000 -- Fri Feb 14 09:03:55 2025
crtime: 0x67af4d4b:bf363200 -- Fri Feb 14 09:03:55 2025
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x5:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  user.job (5) = "lfs.0"
  linkea: idx=0 parent=[0x200000402:0x1:0x0] name='d154.conf-sanity.empty'
BLOCKS:
(0):24783
TOTAL: 1

oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  53387   40755 (2)      0      0    4096 14-Feb-2025 09:03 .
  53383   40755 (18)      0      0    4096 14-Feb-2025 09:03 fid:[0x200000402:0x1:0x0] ..

file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800abd1d000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800abd1d000.idle_timeout=debug
disable quota as required
==pre-rename parent/.. inodes==
144115205289279489 drwxr-xr-x 5 root root 4096 Feb 14 09:03 /mnt/lustre/d154.conf-sanity
144115205289279489 drwxr-xr-x 5 root root 4096 Feb 14 09:03 /mnt/lustre/d154.conf-sanity/d154.conf-sanity.dx_dir/..
144115205289279489 drwxr-xr-x 5 root root 4096 Feb 14 09:03 /mnt/lustre/d154.conf-sanity
144115205289279489 drwxr-xr-x 5 root root 4096 Feb 14 09:03 /mnt/lustre/d154.conf-sanity/d154.conf-sanity.dirent/..
144115205289279489 drwxr-xr-x 5 root root 4096 Feb 14 09:03 /mnt/lustre/d154.conf-sanity
144115205289279489 drwxr-xr-x 5 root root 4096 Feb 14 09:03 /mnt/lustre/d154.conf-sanity/d154.conf-sanity.empty/..
==post-rename parent/.. inodes==
144115205289279490 drwxr-xr-x 5 root root 4096 Feb 14 09:05 /mnt/lustre/d154.conf-sanity.tgt
144115205289279490 drwxr-xr-x 5 root root 4096 Feb 14 09:05 /mnt/lustre/d154.conf-sanity.tgt/d154.conf-sanity.dx_dir/..
144115205289279490 drwxr-xr-x 5 root root 4096 Feb 14 09:05 /mnt/lustre/d154.conf-sanity.tgt
144115205289279490 drwxr-xr-x 5 root root 4096 Feb 14 09:05 /mnt/lustre/d154.conf-sanity.tgt/d154.conf-sanity.dirent/..
144115205289279490 drwxr-xr-x 5 root root 4096 Feb 14 09:05 /mnt/lustre/d154.conf-sanity.tgt
144115205289279490 drwxr-xr-x 5 root root 4096 Feb 14 09:05 /mnt/lustre/d154.conf-sanity.tgt/d154.conf-sanity.empty/..
==target parent FID: fid:[0x200000402:0x2:0x0]==
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
==post-rename .dx_dir should have 'fid:[0x200000402:0x2:0x0] ..'==
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 53329   Type: directory    Mode:  0755   Flags: 0x1000
Generation: 3596354304    Version: 0x00000002:00000001
User:     0   Group:     0   Project:     0   Size: 12288
File ACL: 0
Links: 2   Blockcount: 24
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x67af4d91:00000000 -- Fri Feb 14 09:05:05 2025
 atime: 0x67af4d70:cfa99f00 -- Fri Feb 14 09:04:32 2025
 mtime: 0x67af4d4c:00000000 -- Fri Feb 14 09:03:56 2025
crtime: 0x67af4d70:cfa99f00 -- Fri Feb 14 09:04:32 2025
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x3:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  linkea: idx=0 parent=[0x200000402:0x2:0x0] name='d154.conf-sanity.dx_dir'
BLOCKS:
(0):24207, (1-2):24160-24161
TOTAL: 3

oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  53329   40755 (2)      0      0   12288 14-Feb-2025 09:03 .
  26673   40755 (18)      0      0    4096 14-Feb-2025 09:05 fid:[0x200000402:0x2:0x0] ..
     19  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename103
     20  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename79
     23  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename118
     24  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename89
     25  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename77
     26  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename80
     27  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename122
     29  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename25
     30  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename8
     33  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename6
     34  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename55
     35  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename3
     37  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename92
     38  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename98
     40  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename73
     44  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename90
     45  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename105
     46  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename62
     48  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename43
     50  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename56
     52  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename91
     53  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename70
     55  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename36
     56  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename44
     58  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename116
     60  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename95
     61  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename81
     66  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename33
     70  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename27
     72  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename53
     73  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename41
     74  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename9
     75  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename57
     77  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename61
     79  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename49
     80  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename85
     84  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename10
     87  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename21
     89  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename23
     91  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename58
     96  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename15
     97  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename48
     98  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename114
    105  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename32
    107  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename1
    109  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename72
    111  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename22
    113  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename31
    118  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename112
    119  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename117
    120  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename59
    124  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename0
    127  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename20
    128  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename40
    135  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename13
    136  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename111
    138  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename86
    139  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename69
    145  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename125
     28  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename110
    100  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename96
     92  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename51
    115  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename101
     69  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename99
     68  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename7
    101  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename16
     21  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename100
     76  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename28
     43  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename123
     83  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename5
     39  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename2
     64  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename11
     78  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename66
    102  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename67
     51  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename24
     57  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename47
    116  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename93
    117  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename35
     94  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename14
     47  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename78
    110  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename4
     31  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename65
    103  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename115
     93  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename29
     41  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename76
     85  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename63
     49  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename30
    112  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename37
     62  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename107
     59  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename121
    108  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename17
    114  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename19
     81  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename108
     88  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename42
     22  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename68
    106  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename104
     86  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename39
     67  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename50
     63  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename113
     42  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename71
     71  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename82
     99  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename94
     54  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename106
     32  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename119
     36  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename46
     82  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename54
     95  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename64
     65  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename127
     90  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename52
    104  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename109
    121  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename126
    122  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename124
    123  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename12
    125  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename102
    126  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename120
    129  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename97
    130  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename84
    131  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename75
    132  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename45
    133  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename88
    134  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename18
    137  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename26
    140  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename87
    141  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename38
    142  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename83
    143  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename60
    144  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename34
    146  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename74

==post-rename .dirent should have 'fid:[0x200000402:0x2:0x0] ..'==
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 26671   Type: directory    Mode:  0755   Flags: 0x0
Generation: 3596354433    Version: 0x00000002:00000002
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x67af4d91:00000000 -- Fri Feb 14 09:05:05 2025
 atime: 0x67af4d70:d2c31400 -- Fri Feb 14 09:04:32 2025
 mtime: 0x67af4d4c:00000000 -- Fri Feb 14 09:03:56 2025
crtime: 0x67af4d70:d2c31400 -- Fri Feb 14 09:04:32 2025
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x4:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  linkea: idx=0 parent=[0x200000402:0x2:0x0] name='d154.conf-sanity.dirent'
BLOCKS:
(0):24162
TOTAL: 1

oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  26671   40755 (2)      0      0    4096 14-Feb-2025 09:03 .
  26673   40755 (18)      0      0    4096 14-Feb-2025 09:05 fid:[0x200000402:0x2:0x0] ..
    147  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename1
    148  100644 (1)      0      0       0 14-Feb-2025 09:03 f154.conf-sanity.longfilename0

==post-rename .empty should have 'fid:[0x200000402:0x2:0x0] ..'==
oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
Inode: 26672   Type: directory    Mode:  0755   Flags: 0x0
Generation: 3596354436    Version: 0x00000002:00000003
User:     0   Group:     0   Project:     0   Size: 4096
File ACL: 0
Links: 2   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x67af4d91:00000000 -- Fri Feb 14 09:05:05 2025
 atime: 0x67af4d70:d3001d00 -- Fri Feb 14 09:04:32 2025
 mtime: 0x67af4d4b:00000000 -- Fri Feb 14 09:03:55 2025
crtime: 0x67af4d70:d3001d00 -- Fri Feb 14 09:04:32 2025
Size of extra inode fields: 32
Extended attributes:
  lma: fid=[0x200000402:0x5:0x0] compat=0 incompat=0
  trusted.dmv (48) = d0 0c d3 0c 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
  linkea: idx=0 parent=[0x200000402:0x2:0x0] name='d154.conf-sanity.empty'
BLOCKS:
(0):24163
TOTAL: 1

oleg238-server: debugfs 1.46.2.wc5 (26-Mar-2022)
oleg238-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
  26672   40755 (2)      0      0    4096 14-Feb-2025 09:03 .
  26673   40755 (18)      0      0    4096 14-Feb-2025 09:05 fid:[0x200000402:0x2:0x0] ..

e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8
oleg238-server: e2fsck 1.46.2.wc5 (26-Mar-2022)
oleg238-server: Use max possible thread num: 1 instead
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 3)
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2564: increase inode 209 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2564: increase inode 291 badness 0 to 2 for 10084
[Thread 0] group 1 finished
[Thread 0] group 2 finished
[Thread 0] group 3 finished
[Thread 0] Pass 1: Memory used: 276k/0k (144k/133k), time:  0.00/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 390.63MB/s
[Thread 0] Scanned group range [0, 3), inodes 414
Pass 2: Checking directory structure
Pass 2: Memory used: 276k/0k (98k/179k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 397.30MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 276k/0k (98k/179k), time:  0.01/ 0.01/ 0.00
Pass 3: Memory used: 276k/0k (96k/181k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 276k/0k (68k/209k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 276k/0k (67k/210k), time:  0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 310.95MB/s

         412 inodes used (0.52%, out of 79992)
           2 non-contiguous files (0.5%)
           1 non-contiguous directory (0.2%)
             # of inodes with ind/dind/tind blocks: 0/0/0
       24569 blocks used (49.14%, out of 50000)
           0 bad blocks
           1 large file

         280 regular files
         122 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         402 files
Memory used: 276k/0k (66k/211k), time:  0.02/ 0.01/ 0.00
I/O read: 1MB, write: 0MB, rate: 55.60MB/s
PASS 154 (162s)

== conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:05:40 (1739541940)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg238-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg238-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg238-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
stop mds service on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost2_flakey
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800abd1f000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800abd1f000.idle_timeout=debug
disable quota as required
osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=1
osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=1
Creating to objid 33 on ost lustre-OST0001...
Creating to objid 65 on ost lustre-OST0000...
total: 33 open/close in 0.29 seconds: 113.39 ops/second
total: 33 open/close in 0.31 seconds: 107.63 ops/second
osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=0
osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=0
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
Checking servers environments
Checking clients oleg238-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Starting client oleg238-client.virtnet:  -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre
Started clients oleg238-client.virtnet: 
192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800abd18000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800abd18000.idle_timeout=debug
disable quota as required
osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=1
osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=1
Creating to objid 97 on ost lustre-OST0001...
Creating to objid 97 on ost lustre-OST0000...
total: 33 open/close in 0.33 seconds: 99.58 ops/second
total: 33 open/close in 0.33 seconds: 99.09 ops/second
osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=0
osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=0
PASS 155 (138s)

== conf-sanity test 200a: check CPU partitions =========== 09:07:58 (1739542078)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping client oleg238-client.virtnet /mnt/lustre opts:
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_npartitions=4
libcfs will create CPU partition based on online CPUs
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
../libcfs/libcfs/libcfs options: 'cpu_npartitions=4'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
0	: 0
1	: 1
2	: 2
3	: 3
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
PASS 200a (42s)

== conf-sanity test 200b: set CPU pattern using core selection ========================================================== 09:08:41 (1739542121)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
LNET unconfigure error 22: Invalid argument
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_pattern="0[0-3]"
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_pattern="0[0-3]"'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
/sys/module/libcfs/parameters/cpu_npartitions:0
/sys/module/libcfs/parameters/cpu_pattern:0[0-3]
0	: 0 1 2 3
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
stop mds service on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
modules unloaded.
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_pattern="0[1-2]"
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_pattern="0[1-2]"'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
0	: 1 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
PASS 200b (23s)

== conf-sanity test 200c: set CPU pattern using NUMA node layout ========================================================== 09:09:03 (1739542143)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
LNET unconfigure error 22: Invalid argument
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_pattern="N"
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_pattern="N"'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
/sys/module/libcfs/parameters/cpu_npartitions:0
/sys/module/libcfs/parameters/cpu_pattern:N
0	: 0 1 2 3
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
stop mds service on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
oleg238-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_pattern="0[0-3]"
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_pattern="0[0-3]"'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
stop mds service on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
oleg238-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_pattern="N 0[0]"
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_pattern="N 0[0]"'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
0	: 0 1 2 3
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
PASS 200c (29s)

== conf-sanity test 200d: set CPU pattern to exclude only CPU 1 ========================================================== 09:09:32 (1739542172)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
LNET unconfigure error 22: Invalid argument
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_npartitions=2
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
full_table:
0	: 0 1
1	: 2 3
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
stop mds service on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
oleg238-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_npartitions=2 cpu_pattern="X[1]"
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2 cpu_pattern="X[1]"'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
table with CPU 1 excluded:
/sys/module/libcfs/parameters/cpu_npartitions:2
/sys/module/libcfs/parameters/cpu_pattern:X[1]
0	: 0
1	: 2 3
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
stop mds service on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
oleg238-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_pattern="N"
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_pattern="N"'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
full_table:
0	: 0 1 2 3
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
stop mds service on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
oleg238-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_pattern="N X[1]"
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_pattern="N X[1]"'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
table with CPU 1 excluded:
/sys/module/libcfs/parameters/cpu_npartitions:0
/sys/module/libcfs/parameters/cpu_pattern:N X[1]
0	: 0 2 3
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
PASS 200d (35s)

== conf-sanity test 200e: set CPU pattern using relative core exclusion ========================================================== 09:10:07 (1739542207)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
LNET unconfigure error 22: Invalid argument
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_pattern="N"
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_pattern="N"'
libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
full_table:
0	: 0 1 2 3
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
stop mds service on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
oleg238-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_pattern="N C[0]"
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_pattern="N C[0]"'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
/sys/module/libcfs/parameters/cpu_npartitions:0
/sys/module/libcfs/parameters/cpu_pattern:N C[0]
table with npartitions=2:
0	: 1 2 3
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
stop mds service on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
oleg238-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_npartitions=2
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
full_table:
0	: 0 1
1	: 2 3
umount lustre on /mnt/lustre.....
stop ost1 service on oleg238-server
stop mds service on oleg238-server
stop mds service on oleg238-server
unloading modules via unload_modules_local on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing unload_modules_local
oleg238-server: LNET unconfigure error 22: Invalid argument
modules unloaded.
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=cpu_pattern="C[0]" cpu_npartitions=2
libcfs will create CPU partition based on online CPUs
../libcfs/libcfs/libcfs options: 'cpu_pattern="C[0]" cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
/sys/module/libcfs/parameters/cpu_npartitions:2
/sys/module/libcfs/parameters/cpu_pattern:C[0]
table with npartitions=2:
0	: 1
1	: 3
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:)
PASS 200e (35s)

== conf-sanity test 802a: simulate readonly device ======= 09:10:42 (1739542242)

 SKIP: conf-sanity test_802a ZFS specific test
SKIP 802a (1s)
Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2
oleg238-server: oleg238-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg238-server'
oleg238-server: oleg238-server.virtnet: executing load_modules_local
oleg238-server: Loading modules from /home/green/git/lustre-release/lustre
oleg238-server: detected 4 online CPUs by sysfs
oleg238-server: MODOPTS_LIBCFS=
oleg238-server: Force libcfs to create 2 CPU partitions
oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
== conf-sanity test complete, duration 8773 sec ========== 09:11:02 (1739542262)
conf-sanity: FAIL: test_76a error while apply client_cache_count
=== conf-sanity: start cleanup 09:11:02 (1739542262) ===
=== conf-sanity: finish cleanup 09:11:03 (1739542263) ===