== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:15:51 (1746126951) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre2 Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1550304 .. Extending file size to 4527552 .. failed Extending file size to 6978400 .. Extending file size to 6100960 .. Extending file size to 7598144 .. Extending file size to 7446848 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9022240 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7946624 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8354048 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8408512 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8885536 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 1931 1932 1933 1935 1937 failed 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) 192.168.203.142@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg342-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 8192-9015296 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 8192-9015296 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:06 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 6 ? S 0:01 [ksoftirqd/0] 7 ? S 0:05 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:36 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:13 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:03 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:13 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:02 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:12 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:03 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:11 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 49 ? S 0:00 [kworker/0:1] 50 ? S 0:00 [kworker/1:1] 53 ? S 0:00 [kswapd0] 54 ? SN 0:00 [ksmd] 55 ? SN 0:00 [khugepaged] 56 ? S< 0:00 [nfsiod] 57 ? S< 0:00 [crypto] 65 ? S< 0:00 [kthrotld] 67 ? S< 0:00 [kmpath_rdacd] 68 ? S< 0:00 [kpsmoused] 69 ? S 0:00 [kworker/2:2] 70 ? S< 0:00 [ipv6_addrconf] 83 ? S< 0:00 [deferwq] 290 ? S< 0:00 [ata_sff] 305 ? S 0:00 [scsi_eh_0] 306 ? S< 0:00 [scsi_tmf_0] 308 ? S 0:00 [scsi_eh_1] 309 ? S< 0:00 [scsi_tmf_1] 311 ? S 0:01 [kworker/u8:2] 313 ? S 0:00 [kworker/u8:3] 403 ? S< 0:00 [kworker/3:1H] 404 ? S< 0:00 [kworker/2:1H] 453 ? Ss 0:01 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 456 ? S< 0:01 [nbd0] 487 ? S 0:00 [jbd2/nbd0-8] 488 ? S< 0:00 [ext4-rsv-conver] 570 ? Ss 0:02 /usr/lib/systemd/systemd-journald 605 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 607 ? S 0:00 [kworker/3:2] 732 ? S< 0:00 [kvm-irqfd-clean] 733 ? S< 0:00 [edac-poller] 745 ? S< 0:00 [kworker/0:1H] 779 ? S 0:00 [nfsv4.1-svc] 813 ? Ss 0:01 /usr/lib/systemd/systemd-logind 815 ? Ss 0:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 818 ? Ssl 0:00 /usr/sbin/gssproxy -D 826 ? Ssl 0:01 /usr/sbin/NetworkManager --no-daemon 834 ? Ss 0:00 /usr/sbin/sshd -D 851 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 852 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 853 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 867 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 870 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-09ab79be-487c-40d9-ae34-01c5fcad4280-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 903 ? Ssl 0:02 /usr/sbin/rsyslogd -n 982 ? S< 0:00 [kworker/1:1H] 1133 ? Ss 0:00 sshd: root@pts/0 1177 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1697 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 1698 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg342-client.log 2011 ? S 0:00 [ptlrpc_hr00_000] 2012 ? S 0:00 [ptlrpc_hr01_000] 2013 ? S 0:00 [socknal_cd00] 2014 ? S 0:00 [socknal_cd01] 2015 ? S 0:00 [socknal_cd02] 2016 ? S 0:00 [socknal_cd03] 2017 ? S 0:00 [socknal_reaper] 2018 ? S 0:00 [socknal_sd00_00] 2019 ? S 0:00 [socknal_sd00_01] 2020 ? S 0:19 [socknal_sd01_00] 2021 ? S 0:19 [socknal_sd01_01] 2022 ? S 0:00 [acceptor_001] 2023 ? S 0:00 [monitor_thread] 2024 ? S 0:00 [lnet_discovery] 2025 ? S 0:00 [ptlrpcd_rcv] 2026 ? S 0:06 [ptlrpcd_00_00] 2027 ? S 0:06 [ptlrpcd_00_01] 2028 ? S 0:03 [ptlrpcd_01_00] 2029 ? S 0:03 [ptlrpcd_01_01] 4721 ? S 0:00 /usr/lib/systemd/systemd-udevd 4742 pts/0 R+ 0:00 ps ax 7713 pts/0 S+ 0:01 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 8630 ? S 0:00 [ldlm_cb00_000] 8631 ? S 0:00 [ldlm_cb00_001] 8632 ? S 0:01 [ldlm_cb01_000] 8633 ? S 0:01 [ldlm_cb01_001] 8634 ? S 0:00 [ldlm_cn00_000] 8635 ? S 0:00 [ldlm_cn00_001] 8636 ? S 0:00 [ldlm_cn00_002] 8637 ? S 0:00 [ldlm_cn01_000] 8638 ? S 0:00 [ldlm_cn01_001] 8639 ? S 0:00 [ldlm_cn01_002] 8640 ? S 0:01 [ldlm_bl_01] 8641 ? S 0:01 [ldlm_bl_02] 8642 ? S 0:00 [ldlm_elt] 8643 ? S 0:00 [ll_cfg_requeue] 14253 ? S 0:01 [ldlm_bl_03] 14270 ? S 0:01 [kworker/u8:4] 16449 ? S 0:01 [ldlm_cb01_002] 18369 ? S 0:00 [kworker/0:2] 19075 ? S 0:03 [kworker/2:1] 20403 ? S 0:00 [kworker/1:2] 21071 ? S 0:00 [kworker/2:0] 21247 ? S 0:00 [kworker/0:0] 21399 ? S 0:00 [kworker/1:0] 23794 ? S 0:04 [kworker/3:0] 24139 ? S 0:00 [kworker/3:1] 26339 ? S 0:01 [ldlm_cb01_003] 26525 ? S 0:01 [ldlm_bl_04] 26949 ? S 0:00 [ldlm_cb00_002] 27908 ? S 0:00 [ldlm_bl_05] 29674 ? S 0:00 [kworker/u8:0] 30904 ? S 0:00 [ldlm_bl_06] 30905 ? S 0:00 [ldlm_bl_07] 32472 ? S 0:00 [ldlm_bl_08] sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746127014.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg342-server.1746127014.log.iAjJ89" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg342-server.1746127014.log.MUfmvS" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]