== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:53:23 (1746147203) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1879840 .. Extending file size to 1426400 .. Extending file size to 1530336 .. Extending file size to 7329984 .. Extending file size to 2004096 .. Extending file size to 4249888 .. Extending file size to 8049824 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8556064 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7512960 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9134112 .. Extending file size to 8240576 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8268960 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8890976 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9167008 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 30170 30171 30174 30175 30176 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) 192.168.203.136@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 0-9170944 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-9170944 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:05 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:01 [kworker/u8:0] 6 ? S 0:01 [ksoftirqd/0] 7 ? S 0:02 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:17 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:08 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:01 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:08 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:00 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:06 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:01 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:07 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 49 ? S 0:00 [kworker/0:1] 52 ? S 0:00 [kswapd0] 53 ? SN 0:00 [ksmd] 54 ? SN 0:00 [khugepaged] 55 ? S< 0:00 [nfsiod] 56 ? S< 0:00 [crypto] 64 ? S< 0:00 [kthrotld] 65 ? S 0:00 [kworker/u8:1] 66 ? S< 0:00 [kmpath_rdacd] 67 ? S< 0:00 [kpsmoused] 68 ? S< 0:00 [ipv6_addrconf] 81 ? S< 0:00 [deferwq] 82 ? S 0:00 [kworker/1:2] 275 ? S< 0:00 [ata_sff] 277 ? S 0:00 [scsi_eh_0] 278 ? S< 0:00 [scsi_tmf_0] 280 ? S 0:00 [scsi_eh_1] 281 ? S< 0:00 [scsi_tmf_1] 285 ? S 0:00 [kworker/u8:3] 446 ? Ss 0:00 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 448 ? S< 0:00 [nbd0] 475 ? S 0:00 [jbd2/nbd0-8] 476 ? S< 0:00 [ext4-rsv-conver] 562 ? Ss 0:01 /usr/lib/systemd/systemd-journald 570 ? S 0:01 [kworker/2:2] 594 ? S 0:00 [kworker/3:2] 597 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 711 ? S< 0:00 [kvm-irqfd-clean] 724 ? S< 0:00 [edac-poller] 733 ? S< 0:00 [kworker/0:1H] 734 ? S< 0:00 [kworker/2:1H] 766 ? S< 0:00 [kworker/3:1H] 767 ? S 0:00 [nfsv4.1-svc] 802 ? Ss 0:02 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 807 ? Ss 0:01 /usr/lib/systemd/systemd-logind 809 ? Ssl 0:00 /usr/sbin/NetworkManager --no-daemon 810 ? Ssl 0:00 /usr/sbin/gssproxy -D 823 ? Ss 0:00 /usr/sbin/sshd -D 839 ? S 0:00 [kworker/0:2] 840 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 841 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 843 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 856 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 861 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-ebcb5379-42ce-4960-91f3-266c28c64d93-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 888 ? Ssl 0:01 /usr/sbin/rsyslogd -n 1122 ? Ss 0:00 sshd: root@pts/0 1159 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1608 ? S< 0:00 [kworker/1:1H] 1941 ? S 0:00 [ptlrpc_hr00_000] 1942 ? S 0:00 [ptlrpc_hr01_000] 1943 ? S 0:00 [socknal_cd00] 1944 ? S 0:00 [socknal_cd01] 1945 ? S 0:00 [socknal_cd02] 1946 ? S 0:00 [socknal_cd03] 1947 ? S 0:00 [socknal_reaper] 1948 ? S 0:00 [socknal_sd00_00] 1949 ? S 0:00 [socknal_sd00_01] 1950 ? S 0:16 [socknal_sd01_00] 1951 ? S 0:16 [socknal_sd01_01] 1952 ? S 0:00 [acceptor_001] 1953 ? S 0:00 [monitor_thread] 1954 ? S 0:00 [lnet_discovery] 1955 ? S 0:01 [ptlrpcd_rcv] 1956 ? S 0:05 [ptlrpcd_00_00] 1957 ? S 0:05 [ptlrpcd_00_01] 1958 ? S 0:03 [ptlrpcd_01_00] 1959 ? S 0:03 [ptlrpcd_01_01] 6869 ? S 0:00 [kworker/1:0] 6871 ? S 0:00 [kworker/u8:2] 7036 pts/0 S+ 0:01 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 7249 ? S 0:03 [kworker/2:0] 7591 ? S 0:03 [kworker/3:0] 7631 ? S 0:00 [ldlm_cb00_000] 7632 ? S 0:00 [ldlm_cb00_001] 7633 ? S 0:01 [ldlm_cb01_000] 7634 ? S 0:01 [ldlm_cb01_001] 7635 ? S 0:00 [ldlm_cn00_000] 7636 ? S 0:00 [ldlm_cn00_001] 7637 ? S 0:00 [ldlm_cn00_002] 7638 ? S 0:00 [ldlm_cn01_000] 7639 ? S 0:00 [ldlm_cn01_001] 7640 ? S 0:00 [ldlm_cn01_002] 7641 ? S 0:00 [ldlm_bl_01] 7642 ? S 0:01 [ldlm_bl_02] 7643 ? S 0:00 [ldlm_elt] 7644 ? S 0:00 [ll_cfg_requeue] 12605 ? S 0:01 [ldlm_cb01_002] 14478 ? S 0:01 [ldlm_bl_03] 19584 ? S 0:00 [ldlm_cb00_002] 19601 ? S 0:01 [ldlm_bl_04] 19602 ? S 0:01 [ldlm_bl_05] 27370 ? S 0:00 [ldlm_bl_06] 27372 ? S 0:00 [ldlm_bl_07] 27397 ? S 0:00 [ldlm_bl_08] 29939 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 29942 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg336-client.log 32404 ? S 0:00 /usr/lib/systemd/systemd-udevd 32425 pts/0 R+ 0:00 ps ax sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746147267.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg336-server.1746147267.log.gwvE8z" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg336-server.1746147267.log.cZJdPp" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]