== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:52:35 (1746147155) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre2 Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4850080 .. Extending file size to 3928000 .. failed Extending file size to 6097856 .. Extending file size to 5238688 .. Extending file size to 6600736 .. Extending file size to 8232800 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8669664 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9080864 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8777952 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8959904 .. Waiting 1019 1021 1022 1024 1025 192.168.204.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre2 (opts:) 192.168.204.125@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 16384-9068544 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 16384-9068544 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:05 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:01 [kworker/u8:0] 6 ? S 0:01 [ksoftirqd/0] 7 ? S 0:03 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:27 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:11 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:02 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:10 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:01 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:09 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:02 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:09 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 46 ? S 0:00 [kworker/1:1] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 50 ? S 0:00 [kworker/2:1] 51 ? S 0:04 [kworker/3:1] 53 ? S 0:00 [kswapd0] 54 ? SN 0:00 [ksmd] 55 ? SN 0:00 [khugepaged] 56 ? S< 0:00 [nfsiod] 57 ? S< 0:00 [crypto] 65 ? S< 0:00 [kthrotld] 67 ? S< 0:00 [kmpath_rdacd] 68 ? S< 0:00 [kpsmoused] 70 ? S< 0:00 [ipv6_addrconf] 83 ? S< 0:00 [deferwq] 277 ? S< 0:00 [ata_sff] 281 ? S 0:00 [scsi_eh_0] 282 ? S< 0:00 [scsi_tmf_0] 285 ? S 0:00 [scsi_eh_1] 290 ? S< 0:00 [scsi_tmf_1] 295 ? S 0:00 [kworker/u8:2] 447 ? Ss 0:01 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 449 ? S< 0:00 [nbd0] 477 ? S 0:00 [jbd2/nbd0-8] 479 ? S< 0:00 [ext4-rsv-conver] 483 ? S 0:00 [kworker/0:2] 564 ? Ss 0:02 /usr/lib/systemd/systemd-journald 596 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 628 ? S< 0:00 [kworker/0:1H] 629 ? S< 0:00 [kworker/2:1H] 713 ? S< 0:00 [kvm-irqfd-clean] 726 ? S< 0:00 [edac-poller] 746 ? S 0:00 [kworker/u8:1] 759 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 760 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg425-client.log 766 ? S 0:00 [nfsv4.1-svc] 801 ? Ss 0:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 807 ? Ssl 0:00 /usr/sbin/NetworkManager --no-daemon 808 ? Ss 0:01 /usr/lib/systemd/systemd-logind 809 ? Ssl 0:00 /usr/sbin/gssproxy -D 822 ? Ss 0:00 /usr/sbin/sshd -D 839 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 840 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 842 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 857 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 860 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-cdec1deb-2d58-49de-bb53-1b4cb7c89fe4-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 887 ? Ssl 0:01 /usr/sbin/rsyslogd -n 1120 ? Ss 0:00 sshd: root@pts/0 1159 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1493 ? S< 0:00 [kworker/1:1H] 1985 ? S 0:00 [ptlrpc_hr00_000] 1986 ? S 0:00 [ptlrpc_hr01_000] 1987 ? S 0:00 [socknal_cd00] 1988 ? S 0:00 [socknal_cd01] 1989 ? S 0:00 [socknal_cd02] 1990 ? S 0:00 [socknal_cd03] 1991 ? S 0:00 [socknal_reaper] 1992 ? S 0:00 [socknal_sd00_00] 1993 ? S 0:00 [socknal_sd00_01] 1994 ? S 0:18 [socknal_sd01_00] 1995 ? S 0:18 [socknal_sd01_01] 1996 ? S 0:00 [acceptor_001] 1997 ? S 0:00 [monitor_thread] 1998 ? S 0:00 [lnet_discovery] 1999 ? S 0:01 [ptlrpcd_rcv] 2000 ? S 0:06 [ptlrpcd_00_00] 2001 ? S 0:06 [ptlrpcd_00_01] 2002 ? S 0:03 [ptlrpcd_01_00] 2003 ? S 0:03 [ptlrpcd_01_01] 3058 ? S 0:00 /usr/lib/systemd/systemd-udevd 3079 pts/0 R+ 0:00 ps ax 3393 ? S< 0:00 [kworker/3:1H] 7646 pts/0 S+ 0:01 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 8440 ? S 0:00 [kworker/u8:3] 8476 ? S 0:00 [ldlm_cb00_000] 8477 ? S 0:00 [ldlm_cb00_001] 8478 ? S 0:01 [ldlm_cb01_000] 8479 ? S 0:02 [ldlm_cb01_001] 8480 ? S 0:00 [ldlm_cn00_000] 8481 ? S 0:00 [ldlm_cn00_001] 8482 ? S 0:00 [ldlm_cn00_002] 8483 ? S 0:00 [ldlm_cn01_000] 8484 ? S 0:00 [ldlm_cn01_001] 8485 ? S 0:00 [ldlm_cn01_002] 8486 ? S 0:01 [ldlm_bl_01] 8487 ? S 0:01 [ldlm_bl_02] 8488 ? S 0:00 [ldlm_elt] 8489 ? S 0:00 [ll_cfg_requeue] 16261 ? S 0:01 [ldlm_bl_03] 22416 ? S 0:01 [ldlm_cb01_002] 26339 ? S 0:00 [ldlm_cb00_002] 26824 ? S 0:00 [ldlm_bl_04] 28356 ? S 0:00 [kworker/2:2] 28384 ? S 0:00 [kworker/3:2] 28853 ? S 0:00 [kworker/0:1] 28882 ? S 0:00 [kworker/3:0] 29035 ? S 0:00 [kworker/0:0] 29328 ? S 0:03 [kworker/2:0] 29659 ? S 0:00 [kworker/1:0] 29762 ? S 0:00 [kworker/1:2] 30600 ? S 0:00 [ldlm_bl_05] 30601 ? S 0:00 [ldlm_bl_06] 30748 ? S 0:00 [ldlm_bl_07] 30749 ? S 0:00 [ldlm_bl_08] sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746147219.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg425-server.1746147219.log.I8pIFd" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg425-server.1746147219.log.1XC5Wn" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]