== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:15:57 (1746126957) Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1483072 .. Extending file size to 3362240 .. done Extending file size to 4901376 .. Extending file size to 6607072 .. Extending file size to 7202144 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8658240 .. failed Extending file size to 7884704 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8648576 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8665568 .. Extending file size to 8948736 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8753152 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9032416 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9225696 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 2506 2507 2509 2510 2513 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) 192.168.204.139@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 20480-9207808 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 20480-9207808 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:07 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:02 [kworker/u8:0] 6 ? S 0:01 [ksoftirqd/0] 7 ? S 0:04 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:35 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:13 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:03 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:12 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:02 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:11 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:03 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:11 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 46 ? S 0:02 [kworker/2:1] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 49 ? S 0:00 [kworker/0:1] 51 ? S 0:05 [kworker/3:1] 53 ? S 0:00 [kswapd0] 54 ? SN 0:00 [ksmd] 55 ? SN 0:00 [khugepaged] 56 ? S< 0:00 [nfsiod] 57 ? S< 0:00 [crypto] 65 ? S< 0:00 [kthrotld] 66 ? S 0:00 [kworker/u8:1] 67 ? S< 0:00 [kmpath_rdacd] 68 ? S< 0:00 [kpsmoused] 70 ? S< 0:00 [ipv6_addrconf] 83 ? S< 0:00 [deferwq] 285 ? S< 0:00 [ata_sff] 290 ? S 0:00 [scsi_eh_0] 292 ? S< 0:00 [scsi_tmf_0] 293 ? S 0:00 [scsi_eh_1] 294 ? S< 0:00 [scsi_tmf_1] 302 ? S 0:00 [kworker/u8:3] 449 ? Ss 0:02 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 452 ? S< 0:02 [nbd0] 479 ? S< 0:00 [kworker/0:1H] 482 ? S< 0:00 [kworker/1:1H] 488 ? S 0:00 [jbd2/nbd0-8] 489 ? S< 0:00 [ext4-rsv-conver] 569 ? Ss 0:02 /usr/lib/systemd/systemd-journald 603 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 605 ? S 0:00 [kworker/1:2] 733 ? S< 0:00 [kvm-irqfd-clean] 734 ? S< 0:00 [edac-poller] 782 ? S< 0:00 [kworker/3:1H] 783 ? S 0:00 [nfsv4.1-svc] 785 ? S< 0:00 [kworker/2:1H] 816 ? Ss 0:01 /usr/lib/systemd/systemd-logind 818 ? Ss 0:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 823 ? Ssl 0:01 /usr/sbin/NetworkManager --no-daemon 826 ? Ssl 0:00 /usr/sbin/gssproxy -D 837 ? Ss 0:00 /usr/sbin/sshd -D 855 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 856 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 858 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 868 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 873 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-81e482d8-0363-4884-8927-59548370db15-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 905 ? Ssl 0:02 /usr/sbin/rsyslogd -n 1139 ? Ss 0:00 sshd: root@pts/0 1185 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1287 ? S 0:00 [kworker/u8:2] 2026 ? S 0:00 [ptlrpc_hr00_000] 2027 ? S 0:00 [ptlrpc_hr01_000] 2028 ? S 0:00 [socknal_cd00] 2029 ? S 0:00 [socknal_cd01] 2030 ? S 0:00 [socknal_cd02] 2031 ? S 0:00 [socknal_cd03] 2032 ? S 0:00 [socknal_reaper] 2033 ? S 0:00 [socknal_sd00_00] 2034 ? S 0:00 [socknal_sd00_01] 2035 ? S 0:19 [socknal_sd01_00] 2036 ? S 0:19 [socknal_sd01_01] 2037 ? S 0:00 [acceptor_001] 2038 ? S 0:00 [monitor_thread] 2039 ? S 0:00 [lnet_discovery] 2040 ? S 0:00 [ptlrpcd_rcv] 2041 ? S 0:06 [ptlrpcd_00_00] 2042 ? S 0:06 [ptlrpcd_00_01] 2043 ? S 0:03 [ptlrpcd_01_00] 2044 ? S 0:03 [ptlrpcd_01_01] 2273 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 2274 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg439-client.log 5218 ? S 0:00 /usr/lib/systemd/systemd-udevd 5239 pts/0 R+ 0:00 ps ax 7699 pts/0 S+ 0:02 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 8605 ? S 0:00 [ldlm_cb00_000] 8606 ? S 0:00 [ldlm_cb00_001] 8607 ? S 0:01 [ldlm_cb01_000] 8608 ? S 0:01 [ldlm_cb01_001] 8609 ? S 0:00 [ldlm_cn00_000] 8610 ? S 0:00 [ldlm_cn00_001] 8611 ? S 0:00 [ldlm_cn00_002] 8612 ? S 0:00 [ldlm_cn01_000] 8613 ? S 0:00 [ldlm_cn01_001] 8614 ? S 0:00 [ldlm_cn01_002] 8615 ? S 0:01 [ldlm_bl_01] 8616 ? S 0:01 [ldlm_bl_02] 8617 ? S 0:00 [ldlm_elt] 8618 ? S 0:00 [ll_cfg_requeue] 16393 ? S 0:01 [ldlm_bl_03] 18335 ? S 0:00 [kworker/3:2] 18505 ? S 0:01 [kworker/2:2] 19044 ? S 0:00 [kworker/1:1] 19225 ? S 0:02 [ldlm_cb01_002] 19226 ? S 0:00 [ldlm_cb00_002] 20554 ? S 0:00 [kworker/1:0] 20706 ? S 0:00 [kworker/2:0] 21548 ? S 0:00 [kworker/0:2] 22013 ? S 0:00 [kworker/3:0] 22466 ? S 0:00 [kworker/0:0] 26667 ? S 0:01 [ldlm_bl_04] 31560 ? S 0:00 [ldlm_bl_05] 31726 ? S 0:00 [ldlm_bl_06] 31727 ? S 0:00 [ldlm_bl_07] 32059 ? S 0:00 [ldlm_bl_08] sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746127020.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg439-server.1746127020.log.zFsrKk" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg439-server.1746127020.log.cANRrR" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]