== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:02:24 (1746136944) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre2 Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6072928 .. Extending file size to 636256 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1737216 .. failed Extending file size to 6339520 .. Extending file size to 3242336 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7406656 .. Extending file size to 7277760 .. Extending file size to 8236384 .. Extending file size to 8315232 .. Extending file size to 8559104 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8732352 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9023424 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9140960 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9117728 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9275040 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 1143 1144 1145 1146 1147 192.168.203.149@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg349-client.virtnet /mnt/lustre2 (opts:) 192.168.203.149@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg349-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 20480-9256960 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 20480-9256960 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:05 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:00 [kworker/u8:0] 6 ? S 0:01 [ksoftirqd/0] 7 ? S 0:02 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:26 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:09 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:02 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:09 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:01 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 24 ? S 0:00 [kworker/2:0] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:10 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:02 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:10 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 49 ? S 0:00 [kworker/2:1] 50 ? S 0:00 [kworker/3:1] 52 ? S 0:00 [kswapd0] 53 ? SN 0:00 [ksmd] 54 ? SN 0:00 [khugepaged] 55 ? S< 0:00 [nfsiod] 56 ? S< 0:00 [crypto] 64 ? S< 0:00 [kthrotld] 65 ? S 0:01 [kworker/u8:1] 66 ? S< 0:00 [kmpath_rdacd] 67 ? S< 0:00 [kpsmoused] 68 ? S< 0:00 [ipv6_addrconf] 81 ? S< 0:00 [deferwq] 119 ? S 0:04 [kworker/1:1] 273 ? S< 0:00 [ata_sff] 276 ? S 0:00 [scsi_eh_0] 277 ? S< 0:00 [scsi_tmf_0] 285 ? S 0:00 [scsi_eh_1] 288 ? S< 0:00 [scsi_tmf_1] 449 ? Ss 0:01 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 451 ? S< 0:00 [nbd0] 479 ? S 0:00 [jbd2/nbd0-8] 480 ? S< 0:00 [ext4-rsv-conver] 565 ? Ss 0:02 /usr/lib/systemd/systemd-journald 599 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 671 ? S 0:02 [kworker/0:2] 722 ? S< 0:00 [kvm-irqfd-clean] 723 ? S 0:00 [kworker/1:2] 728 ? S< 0:00 [kworker/0:1H] 729 ? S< 0:00 [kworker/3:1H] 730 ? S< 0:00 [edac-poller] 770 ? S 0:00 [nfsv4.1-svc] 804 ? Ss 0:01 /usr/lib/systemd/systemd-logind 807 ? Ss 0:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 810 ? Ssl 0:00 /usr/sbin/gssproxy -D 818 ? Ssl 0:00 /usr/sbin/NetworkManager --no-daemon 824 ? Ss 0:00 /usr/sbin/sshd -D 842 ? S 0:00 [kworker/3:2] 844 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 845 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 846 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 860 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 865 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-b0c1f5f0-8153-48f1-acc0-e0bb2d073133-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 889 ? S 0:00 [kworker/0:1] 892 ? Ssl 0:01 /usr/sbin/rsyslogd -n 903 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 906 pts/0 R+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg349-client.log 1066 ? S< 0:00 [kworker/1:1H] 1129 ? Ss 0:00 sshd: root@pts/0 1164 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1989 ? S 0:00 [ptlrpc_hr00_000] 1990 ? S 0:00 [ptlrpc_hr01_000] 1991 ? S 0:00 [socknal_cd00] 1992 ? S 0:00 [socknal_cd01] 1993 ? S 0:00 [socknal_cd02] 1994 ? S 0:00 [socknal_cd03] 1995 ? S 0:00 [socknal_reaper] 1996 ? S 0:16 [socknal_sd00_00] 1997 ? S 0:16 [socknal_sd00_01] 1998 ? S 0:00 [socknal_sd01_00] 1999 ? S 0:00 [socknal_sd01_01] 2000 ? S 0:00 [acceptor_001] 2001 ? S 0:00 [monitor_thread] 2002 ? S 0:00 [lnet_discovery] 2003 ? S 0:00 [ptlrpcd_rcv] 2004 ? S 0:03 [ptlrpcd_00_00] 2005 ? S 0:03 [ptlrpcd_00_01] 2006 ? S 0:06 [ptlrpcd_01_00] 2007 ? S 0:06 [ptlrpcd_01_01] 2652 ? S< 0:00 [kworker/2:1H] 3260 ? S 0:00 [ldlm_bl_08] 3261 ? S 0:00 [ldlm_bl_09] 3427 ? S 0:00 /usr/lib/systemd/systemd-udevd 3449 pts/0 R+ 0:00 ps ax 7630 pts/0 S+ 0:01 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 7730 ? S 0:00 [kworker/u8:3] 8505 ? S 0:01 [ldlm_cb00_000] 8506 ? S 0:01 [ldlm_cb00_001] 8507 ? S 0:00 [ldlm_cb01_000] 8508 ? S 0:00 [ldlm_cb01_001] 8509 ? S 0:00 [ldlm_cn00_000] 8510 ? S 0:00 [ldlm_cn00_001] 8511 ? S 0:00 [ldlm_cn00_002] 8512 ? S 0:00 [ldlm_cn01_000] 8513 ? S 0:00 [ldlm_cn01_001] 8514 ? S 0:00 [ldlm_cn01_002] 8515 ? S 0:01 [ldlm_bl_01] 8516 ? S 0:01 [ldlm_bl_02] 8517 ? S 0:00 [ldlm_elt] 8518 ? S 0:00 [ll_cfg_requeue] 9252 ? S 0:02 [ldlm_cb00_002] 12137 ? S 0:01 [ldlm_bl_03] 26462 ? S 0:01 [ldlm_bl_04] 29197 ? S 0:00 [kworker/2:2] 30535 ? S 0:00 [ldlm_bl_05] 30536 ? S 0:00 [ldlm_bl_06] 30549 ? S 0:00 [ldlm_cb01_002] 30714 ? S 0:00 [ldlm_bl_07] 31164 ? S 0:00 [kworker/0:0] 31334 ? S 0:00 [kworker/1:0] sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746137009.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg349-server.1746137009.log.fXjsvj" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg349-server.1746137009.log.pNHGd5" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]