== conf-sanity test 35a: Reconnect to the last active server first ========================================================== 14:54:07 (1789584847)
start mds service on oleg251-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep 16 09:22 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg251-server'
oleg251-server: oleg251-server.virtnet: executing load_modules_local
oleg251-server: kmemleak disabled
oleg251-server: Loading modules from /home/green/git/lustre-release/lustre
oleg251-server: detected 4 online CPUs by sysfs
oleg251-server: MODOPTS_LIBCFS=
oleg251-server: Force libcfs to create 2 CPU partitions
oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg251-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg251-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg251-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg251-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1
Started lustre-OST0000
oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg251-client.virtnet:  -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre
debug=ha
Set up a fake failnode for the MDS
Wait for RECONNECT_INTERVAL seconds (10s)
conf-sanity.sh test_35a 2026-09-1614h55m57s
Stopping the MDT: lustre-MDT0000
stop mds service on oleg251-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server
Restarting the MDT: lustre-MDT0000
start mds service on oleg251-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1
Started lustre-MDT0000
Wait for df (40625) ... 
done
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout
Debug log: 136 lines, 136 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg251-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg251-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server
stop mds service on oleg251-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server
stop mds service on oleg251-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server
unloading modules via unload_modules_local on: 'oleg251-server'
oleg251-server: oleg251-server.virtnet: executing unload_modules_local
oleg251-server: kmemleak disabled
modules unloaded.
oleg251-server: 
oleg251-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg251-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19
oleg251-server: 
oleg251-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg251-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19
oleg251-server: 
oleg251-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg251-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19
oleg251-server: 
oleg251-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg251-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19
tunefs.lustre failed, reformatting instead
Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2
oleg251-server: oleg251-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1931560 Sep 16 09:22 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg251-server'
oleg251-server: oleg251-server.virtnet: executing load_modules_local
oleg251-server: kmemleak disabled
oleg251-server: Loading modules from /home/green/git/lustre-release/lustre
oleg251-server: detected 4 online CPUs by sysfs
oleg251-server: MODOPTS_LIBCFS=
oleg251-server: Force libcfs to create 2 CPU partitions
oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg251-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg251-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg251-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg251-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg251-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg251-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server
stop mds service on oleg251-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server
stop mds service on oleg251-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server