== conf-sanity test 35a: Reconnect to the last active server first ========================================================== 23:11:53 (1717557113)
start mds service on oleg157-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg157-server'
oleg157-server: oleg157-server.virtnet: executing load_modules_local
oleg157-server: Loading modules from /home/green/git/lustre-release/lustre
oleg157-server: detected 4 online CPUs by sysfs
oleg157-server: Force libcfs to create 2 CPU partitions
oleg157-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg157-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg157-server
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg157-server
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Started lustre-OST0000
oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg157-client.virtnet:  -o user_xattr,flock oleg157-server@tcp:/lustre /mnt/lustre
debug=ha
Set up a fake failnode for the MDS
Wait for RECONNECT_INTERVAL seconds (10s)
error: couldn't open trace file
conf-sanity.sh test_35a 2024-06-0423h12m30s
Stopping the MDT: lustre-MDT0000
stop mds service on oleg157-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server
Restarting the MDT: lustre-MDT0000
start mds service on oleg157-server
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Started lustre-MDT0000
Wait for df (31101) ... 
done
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout
error: couldn't open trace file
error: couldn't open trace file
awk: fatal: cannot open file `/tmp/lustre-log-test_35a.log' for reading (No such file or directory)
 conf-sanity test_35a: @@@@@@ FAIL: Tried to connect to  not last active server 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7066:error()
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3268:test_35a()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7411:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7474:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7282:run_test()
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3273:main()
Dumping lctl log to /tmp/testlogs//conf-sanity.test_35a.*.1717557165.log
oleg157-client: error: couldn't open trace file
oleg157-client: error: couldn't open trace file
oleg157-server: error: couldn't open trace file
oleg157-server: error: couldn't open trace file
rsync: chown "/tmp/testlogs/.conf-sanity.test_35a.debug_log.oleg157-server.1717557165.log.NCrGev" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.conf-sanity.test_35a.dmesg.oleg157-server.1717557165.log.sRZcWW" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]