== conf-sanity test 35a: Reconnect to the last active server first ========================================================== 07:25:22 (1764937522)
start mds service on oleg303-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg303-server'
oleg303-server: oleg303-server.virtnet: executing load_modules_local
oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg303-server: Loading modules from /home/green/git/lustre-release/lustre
oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg303-server: detected 4 online CPUs by sysfs
oleg303-server: Force libcfs to create 2 CPU partitions
oleg303-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg303-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg303-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-OST0000
oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg303-client.virtnet:  -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre
debug=ha
Set up a fake failnode for the MDS
Wait for RECONNECT_INTERVAL seconds (10s)
conf-sanity.sh test_35a 2025-12-05 7h26m39s
Stopping the MDT: lustre-MDT0000
stop mds service on oleg303-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server
Restarting the MDT: lustre-MDT0000
start mds service on oleg303-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
Wait for df (27302) ... 
done
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout
Debug log: 103 lines, 103 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg303-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg303-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server
stop mds service on oleg303-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg303-server'
oleg303-server: oleg303-server.virtnet: executing unload_modules_local
oleg303-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg303-server: LNET ready to unload
oleg303-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
oleg303-server: 
oleg303-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg303-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg303-server: 
oleg303-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg303-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg303-server: 
oleg303-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg303-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg303-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg303-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2
oleg303-server: oleg303-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg303-server'
oleg303-server: oleg303-server.virtnet: executing load_modules_local
oleg303-server: Loading modules from /home/green/git/lustre-release/lustre
oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg303-server: detected 4 online CPUs by sysfs
oleg303-server: Force libcfs to create 2 CPU partitions
oleg303-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg303-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg303-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg303-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg303-server: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg303-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg303-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server
stop mds service on oleg303-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server