== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:15:54 (1746126954) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre2 Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4846944 .. Extending file size to 3668160 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8337632 .. Extending file size to 5589472 .. Extending file size to 5784320 .. Extending file size to 6736704 .. Extending file size to 7380160 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8650944 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8473920 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8919424 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8857088 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8902368 .. failed Extending file size to 9319456 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 29747 29748 29751 29752 29753 192.168.201.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre2 (opts:) 192.168.201.132@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 0-9322496 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-9322496 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:06 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:01 [kworker/u8:0] 6 ? S 0:02 [ksoftirqd/0] 7 ? S 0:05 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:36 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:13 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:04 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:13 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:02 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:11 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? R 0:03 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:11 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 46 ? S 0:00 [kworker/1:1] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 49 ? S 0:04 [kworker/2:1] 53 ? S 0:00 [kswapd0] 54 ? SN 0:00 [ksmd] 55 ? SN 0:00 [khugepaged] 56 ? S< 0:00 [nfsiod] 57 ? S< 0:00 [crypto] 65 ? S< 0:00 [kthrotld] 67 ? S< 0:00 [kmpath_rdacd] 68 ? S< 0:00 [kpsmoused] 70 ? S< 0:00 [ipv6_addrconf] 83 ? S< 0:00 [deferwq] 127 ? S 0:01 [kworker/3:2] 293 ? S< 0:00 [ata_sff] 303 ? S 0:00 [scsi_eh_0] 305 ? S< 0:00 [scsi_tmf_0] 306 ? S 0:00 [scsi_eh_1] 309 ? S< 0:00 [scsi_tmf_1] 314 ? S 0:00 [kworker/u8:3] 452 ? Ss 0:02 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 454 ? S< 0:02 [nbd0] 483 ? S 0:00 [jbd2/nbd0-8] 485 ? S< 0:00 [ext4-rsv-conver] 525 ? S< 0:00 [kworker/2:1H] 526 ? S< 0:00 [kworker/0:1H] 571 ? Ss 0:02 /usr/lib/systemd/systemd-journald 604 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 726 ? S< 0:00 [kvm-irqfd-clean] 733 ? S< 0:00 [edac-poller] 744 ? S< 0:00 [kworker/3:1H] 784 ? S 0:00 [nfsv4.1-svc] 819 ? Ss 0:01 /usr/lib/systemd/systemd-logind 821 ? Ss 0:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 824 ? Ssl 0:00 /usr/sbin/gssproxy -D 832 ? Ssl 0:01 /usr/sbin/NetworkManager --no-daemon 848 ? Ss 0:00 /usr/sbin/sshd -D 864 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 867 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 868 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 882 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 889 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-e078118f-4129-4075-86e5-754bd02f6347-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 916 ? Ssl 0:02 /usr/sbin/rsyslogd -n 966 ? S< 0:00 [kworker/1:1H] 1150 ? Ss 0:00 sshd: root@pts/0 1195 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1984 ? S 0:00 [ptlrpc_hr00_000] 1985 ? S 0:00 [ptlrpc_hr01_000] 1986 ? S 0:00 [socknal_cd00] 1987 ? S 0:00 [socknal_cd01] 1988 ? S 0:00 [socknal_cd02] 1989 ? S 0:00 [socknal_cd03] 1990 ? S 0:00 [socknal_reaper] 1991 ? S 0:00 [socknal_sd00_00] 1992 ? S 0:00 [socknal_sd00_01] 1993 ? S 0:18 [socknal_sd01_00] 1994 ? S 0:18 [socknal_sd01_01] 1995 ? S 0:00 [acceptor_001] 1996 ? S 0:00 [monitor_thread] 1997 ? S 0:00 [lnet_discovery] 1998 ? S 0:00 [ptlrpcd_rcv] 1999 ? S 0:06 [ptlrpcd_00_00] 2000 ? S 0:06 [ptlrpcd_00_01] 2001 ? S 0:03 [ptlrpcd_01_00] 2002 ? S 0:03 [ptlrpcd_01_01] 6990 ? S 0:00 [kworker/u8:4] 7081 pts/0 S+ 0:02 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 7745 ? S 0:00 [ldlm_cb00_000] 7746 ? S 0:00 [ldlm_cb00_001] 7747 ? S 0:01 [ldlm_cb01_000] 7748 ? S 0:01 [ldlm_cb01_001] 7749 ? S 0:00 [ldlm_cn00_000] 7750 ? S 0:00 [ldlm_cn00_001] 7751 ? S 0:00 [ldlm_cn00_002] 7752 ? S 0:00 [ldlm_cn01_000] 7753 ? S 0:00 [ldlm_cn01_001] 7754 ? S 0:00 [ldlm_cn01_002] 7755 ? S 0:01 [ldlm_bl_01] 7756 ? S 0:01 [ldlm_bl_02] 7757 ? S 0:00 [ldlm_elt] 7758 ? S 0:00 [ll_cfg_requeue] 15478 ? S 0:01 [ldlm_bl_03] 15479 ? S 0:01 [ldlm_cb01_002] 17069 ? S 0:03 [kworker/3:0] 19386 ? S 0:00 [kworker/0:1] 21581 ? S 0:00 [kworker/2:2] 21913 ? S 0:00 [kworker/1:0] 23743 ? S 0:00 [kworker/0:0] 23763 ? S 0:01 [ldlm_bl_04] 24695 ? S 0:00 [kworker/1:3] 24812 ? S 0:00 [ldlm_cb00_002] 25774 ? S 0:00 [kworker/2:0] 27090 ? S 0:00 [ldlm_bl_05] 27091 ? S 0:00 [ldlm_bl_06] 27092 ? S 0:00 [ldlm_bl_07] 27093 ? S 0:00 [ldlm_bl_08] 29517 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 29518 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg132-client.log 31550 ? S 0:00 /usr/lib/systemd/systemd-udevd 31572 pts/0 R+ 0:00 ps ax 32756 ? S 0:00 [kworker/u8:2] sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746127019.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg132-server.1746127019.log.TR5CdP" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg132-server.1746127019.log.xe82gp" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]