== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:35:18 (1746131718) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4153856 .. Extending file size to 8900128 .. Extending file size to 4728768 .. Extending file size to 5280448 .. Extending file size to 5609568 .. Extending file size to 8715712 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9131360 .. failed Extending file size to 9308672 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9260032 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 30028 30029 30030 30031 30032 failed 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) 192.168.202.151@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 0-9310208 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-9310208 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:05 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 3 ? S 0:00 [kworker/0:0] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:01 [kworker/u8:0] 6 ? S 0:01 [ksoftirqd/0] 7 ? S 0:03 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:27 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:11 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:02 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:11 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:01 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:09 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:02 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:09 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 51 ? S 0:00 [kswapd0] 52 ? SN 0:00 [ksmd] 53 ? SN 0:00 [khugepaged] 54 ? S< 0:00 [nfsiod] 55 ? S< 0:00 [crypto] 63 ? S< 0:00 [kthrotld] 66 ? S< 0:00 [kmpath_rdacd] 68 ? S< 0:00 [kpsmoused] 69 ? S< 0:00 [ipv6_addrconf] 83 ? S< 0:00 [deferwq] 276 ? S< 0:00 [ata_sff] 287 ? S 0:00 [scsi_eh_0] 288 ? S< 0:00 [scsi_tmf_0] 289 ? S 0:00 [scsi_eh_1] 290 ? S< 0:00 [scsi_tmf_1] 295 ? S 0:00 [kworker/u8:2] 447 ? Ss 0:01 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 449 ? S< 0:00 [nbd0] 451 ? S< 0:00 [kworker/0:1H] 480 ? S 0:00 [jbd2/nbd0-8] 481 ? S< 0:00 [ext4-rsv-conver] 541 ? S 0:00 [kworker/1:2] 566 ? Ss 0:02 /usr/lib/systemd/systemd-journald 582 ? S 0:03 [kworker/2:2] 599 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 713 ? S< 0:00 [kvm-irqfd-clean] 724 ? S< 0:00 [edac-poller] 767 ? S< 0:00 [kworker/3:1H] 768 ? S 0:00 [nfsv4.1-svc] 801 ? Ss 0:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 806 ? Ssl 0:00 /usr/sbin/NetworkManager --no-daemon 808 ? Ss 0:01 /usr/lib/systemd/systemd-logind 809 ? Ssl 0:00 /usr/sbin/gssproxy -D 821 ? Ss 0:00 /usr/sbin/sshd -D 842 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 843 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 844 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 855 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 859 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-5f1bb4a9-0508-4689-85c5-8c3907d50f74-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 890 ? Ssl 0:01 /usr/sbin/rsyslogd -n 894 ? S 0:00 [kworker/3:2] 1066 ? S< 0:00 [kworker/2:1H] 1067 ? S< 0:00 [kworker/1:1H] 1122 ? Ss 0:00 sshd: root@pts/0 1161 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1941 ? S 0:00 [ptlrpc_hr00_000] 1942 ? S 0:00 [ptlrpc_hr01_000] 1943 ? S 0:00 [socknal_cd00] 1944 ? S 0:00 [socknal_cd01] 1945 ? S 0:00 [socknal_cd02] 1946 ? S 0:00 [socknal_cd03] 1947 ? S 0:00 [socknal_reaper] 1948 ? S 0:00 [socknal_sd00_00] 1949 ? S 0:00 [socknal_sd00_01] 1950 ? S 0:17 [socknal_sd01_00] 1951 ? S 0:17 [socknal_sd01_01] 1952 ? S 0:00 [acceptor_001] 1953 ? S 0:00 [monitor_thread] 1954 ? S 0:00 [lnet_discovery] 1955 ? S 0:00 [ptlrpcd_rcv] 1956 ? S 0:05 [ptlrpcd_00_00] 1957 ? S 0:05 [ptlrpcd_00_01] 1958 ? S 0:03 [ptlrpcd_01_00] 1959 ? S 0:02 [ptlrpcd_01_01] 6907 ? S 0:00 [kworker/u8:4] 7029 pts/0 S+ 0:02 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 7667 ? S 0:00 [ldlm_cb00_000] 7668 ? S 0:00 [ldlm_cb00_001] 7669 ? S 0:01 [ldlm_cb01_000] 7670 ? S 0:01 [ldlm_cb01_001] 7671 ? S 0:00 [ldlm_cn00_000] 7672 ? S 0:00 [ldlm_cn00_001] 7673 ? S 0:00 [ldlm_cn00_002] 7674 ? S 0:00 [ldlm_cn01_000] 7675 ? S 0:00 [ldlm_cn01_001] 7676 ? S 0:00 [ldlm_cn01_002] 7677 ? S 0:01 [ldlm_bl_01] 7678 ? S 0:01 [ldlm_bl_02] 7679 ? S 0:00 [ldlm_elt] 7680 ? S 0:00 [ll_cfg_requeue] 13795 ? S 0:01 [ldlm_bl_03] 15412 ? S 0:01 [ldlm_cb01_002] 23640 ? S 0:00 [ldlm_cb00_002] 23852 ? S 0:01 [ldlm_bl_04] 25474 ? S 0:00 [kworker/3:1] 25478 ? S 0:00 [kworker/1:1] 26784 ? S 0:00 [kworker/0:2] 27021 ? S 0:00 [ldlm_bl_05] 27022 ? S 0:00 [ldlm_bl_06] 28028 ? S 0:00 [ldlm_bl_07] 28029 ? S 0:00 [ldlm_bl_08] 28482 ? S 0:00 [kworker/0:1] 28800 ? S 0:03 [kworker/3:0] 29282 ? S 0:00 [kworker/1:0] 29603 ? S 0:01 [kworker/2:0] 29654 ? S 0:00 [kworker/u8:1] 29797 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 29798 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg251-client.log 32333 ? S 0:00 /usr/lib/systemd/systemd-udevd 32355 pts/0 R+ 0:00 ps ax sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746131783.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg251-server.1746131783.log.FJzin7" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg251-server.1746131783.log.qtwM12" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]