== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:55:51 (1746132951) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre2 Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4211104 .. Extending file size to 2620096 .. done Extending file size to 6197536 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6791648 .. Extending file size to 7092096 .. Extending file size to 6539168 .. Extending file size to 9125056 .. Extending file size to 8170720 .. Extending file size to 8943648 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9363776 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9256256 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 32660 32661 32662 32663 32670 failed 192.168.201.158@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg158-client.virtnet /mnt/lustre2 (opts:) 192.168.201.158@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg158-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 36864-9330688 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 36864-9330688 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:07 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:01 [kworker/u8:0] 6 ? S 0:02 [ksoftirqd/0] 7 ? S 0:05 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? R 0:40 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:14 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:04 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 17 ? S 0:00 [kworker/1:0] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:14 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:02 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:12 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:03 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:12 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 46 ? S 0:00 [kworker/0:1] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 51 ? S 0:00 [kswapd0] 52 ? SN 0:00 [ksmd] 53 ? SN 0:00 [khugepaged] 54 ? S< 0:00 [nfsiod] 55 ? S< 0:00 [crypto] 63 ? S< 0:00 [kthrotld] 65 ? S< 0:00 [kmpath_rdacd] 66 ? S< 0:00 [kpsmoused] 67 ? S< 0:00 [ipv6_addrconf] 81 ? S< 0:00 [deferwq] 277 ? S< 0:00 [ata_sff] 287 ? S 0:00 [scsi_eh_0] 289 ? S< 0:00 [scsi_tmf_0] 293 ? S 0:00 [scsi_eh_1] 294 ? S< 0:00 [scsi_tmf_1] 300 ? S 0:01 [kworker/u8:2] 448 ? Ss 0:01 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 450 ? S< 0:01 [nbd0] 477 ? S 0:00 [jbd2/nbd0-8] 478 ? S< 0:00 [ext4-rsv-conver] 545 ? S< 0:00 [kworker/0:1H] 565 ? Ss 0:02 /usr/lib/systemd/systemd-journald 581 ? S 0:03 [kworker/2:2] 595 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 714 ? S< 0:00 [kvm-irqfd-clean] 727 ? S< 0:00 [edac-poller] 767 ? S< 0:00 [kworker/3:1H] 768 ? S 0:00 [nfsv4.1-svc] 796 ? S 0:00 [kworker/1:2] 804 ? Ss 0:04 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 809 ? Ssl 0:01 /usr/sbin/NetworkManager --no-daemon 811 ? Ss 0:02 /usr/lib/systemd/systemd-logind 812 ? Ssl 0:00 /usr/sbin/gssproxy -D 824 ? Ss 0:00 /usr/sbin/sshd -D 842 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 843 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 844 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 860 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 863 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-faffe89b-e9f1-4ec5-878e-3e72cac30aec-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 890 ? Ssl 0:02 /usr/sbin/rsyslogd -n 995 ? S< 0:00 [kworker/2:1H] 1126 ? Ss 0:00 sshd: root@pts/0 1165 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1483 ? S< 0:00 [kworker/1:1H] 1762 ? S 0:00 [kworker/u8:1] 1994 ? S 0:00 [ptlrpc_hr00_000] 1995 ? S 0:00 [ptlrpc_hr01_000] 1996 ? S 0:00 [socknal_cd00] 1997 ? S 0:00 [socknal_cd01] 1998 ? S 0:00 [socknal_cd02] 1999 ? S 0:00 [socknal_cd03] 2000 ? S 0:00 [socknal_reaper] 2001 ? S 0:00 [socknal_sd00_00] 2002 ? S 0:00 [socknal_sd00_01] 2003 ? S 0:19 [socknal_sd01_00] 2004 ? S 0:19 [socknal_sd01_01] 2005 ? S 0:00 [acceptor_001] 2006 ? S 0:00 [monitor_thread] 2007 ? S 0:00 [lnet_discovery] 2008 ? S 0:00 [ptlrpcd_rcv] 2009 ? S 0:06 [ptlrpcd_00_00] 2010 ? S 0:06 [ptlrpcd_00_01] 2011 ? S 0:03 [ptlrpcd_01_00] 2012 ? S 0:03 [ptlrpcd_01_01] 2397 ? S 0:00 /usr/lib/systemd/systemd-udevd 2419 pts/0 R+ 0:00 ps ax 7674 pts/0 S+ 0:02 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 8580 ? S 0:00 [ldlm_cb00_000] 8581 ? S 0:00 [ldlm_cb00_001] 8582 ? S 0:02 [ldlm_cb01_000] 8583 ? S 0:02 [ldlm_cb01_001] 8584 ? S 0:00 [ldlm_cn00_000] 8585 ? S 0:00 [ldlm_cn00_001] 8586 ? S 0:00 [ldlm_cn00_002] 8587 ? S 0:00 [ldlm_cn01_000] 8588 ? S 0:00 [ldlm_cn01_001] 8589 ? S 0:00 [ldlm_cn01_002] 8590 ? S 0:01 [ldlm_bl_01] 8591 ? S 0:01 [ldlm_bl_02] 8592 ? S 0:00 [ldlm_elt] 8593 ? S 0:00 [ll_cfg_requeue] 14147 ? S 0:01 [ldlm_bl_03] 17074 ? S 0:01 [ldlm_cb01_002] 18492 ? S 0:00 [kworker/3:1] 19032 ? S 0:00 [kworker/0:2] 23299 ? S 0:00 [kworker/2:1] 23735 ? S 0:03 [kworker/3:0] 24901 ? S 0:00 [kworker/2:0] 27385 ? S 0:00 [ldlm_bl_04] 27541 ? S 0:00 [ldlm_cb00_002] 30083 ? S 0:00 [ldlm_bl_05] 30084 ? S 0:00 [ldlm_bl_06] 30085 ? S 0:00 [ldlm_bl_07] 30532 ? S 0:00 [ldlm_bl_08] 30533 ? S 0:00 [ldlm_bl_09] 32411 ? S 0:00 [kworker/u8:3] 32424 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 32425 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg158-client.log sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746133016.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg158-server.1746133016.log.wQLQfS" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg158-server.1746133016.log.AdBA4w" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]