== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:56:31 (1746143791) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7114624 .. Extending file size to 492992 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3774944 .. Extending file size to 5038464 .. Extending file size to 7376480 .. Extending file size to 6826496 .. failed Extending file size to 8641056 .. Extending file size to 7597440 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7874464 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8171168 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8379712 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8682176 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8999296 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8793280 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8859008 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9045536 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 749 750 752 753 755 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) 192.168.202.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg203-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 4096-9043968 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 4096-9043968 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:06 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 6 ? S 0:02 [ksoftirqd/0] 7 ? S 0:05 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:37 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:14 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:03 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:13 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:02 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:12 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:03 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:11 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 46 ? S 0:01 [kworker/2:1] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 49 ? S 0:00 [kworker/0:1] 50 ? S 0:00 [kworker/1:1] 51 ? S 0:04 [kworker/3:1] 53 ? S 0:00 [kswapd0] 54 ? SN 0:00 [ksmd] 55 ? SN 0:00 [khugepaged] 56 ? S< 0:00 [nfsiod] 57 ? S< 0:00 [crypto] 65 ? S< 0:00 [kthrotld] 66 ? S 0:01 [kworker/u8:1] 67 ? S< 0:00 [kmpath_rdacd] 68 ? S< 0:00 [kpsmoused] 69 ? S< 0:00 [ipv6_addrconf] 83 ? S< 0:00 [deferwq] 274 ? S< 0:00 [ata_sff] 277 ? S 0:00 [scsi_eh_0] 278 ? S< 0:00 [scsi_tmf_0] 279 ? S 0:00 [scsi_eh_1] 280 ? S< 0:00 [scsi_tmf_1] 282 ? S 0:01 [kworker/u8:2] 448 ? Ss 0:01 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 451 ? S< 0:01 [nbd0] 478 ? S 0:00 [jbd2/nbd0-8] 479 ? S< 0:00 [ext4-rsv-conver] 499 ? S 0:00 [kworker/u8:0] 511 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 512 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg203-client.log 565 ? Ss 0:02 /usr/lib/systemd/systemd-journald 597 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 603 ? S< 0:00 [kworker/0:1H] 713 ? S< 0:00 [kvm-irqfd-clean] 717 ? S< 0:00 [edac-poller] 768 ? S< 0:00 [kworker/3:1H] 769 ? S 0:00 [nfsv4.1-svc] 806 ? Ss 0:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 809 ? Ssl 0:00 /usr/sbin/gssproxy -D 817 ? Ssl 0:01 /usr/sbin/NetworkManager --no-daemon 818 ? Ss 0:01 /usr/lib/systemd/systemd-logind 826 ? Ss 0:00 /usr/sbin/sshd -D 842 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 843 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 845 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 860 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 862 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-64c743fc-cbc9-44ab-8c33-bc1eb48aabef-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 892 ? Ssl 0:01 /usr/sbin/rsyslogd -n 1067 ? S< 0:00 [kworker/2:1H] 1125 ? Ss 0:00 sshd: root@pts/0 1163 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1993 ? S 0:00 [ptlrpc_hr00_000] 1994 ? S 0:00 [ptlrpc_hr01_000] 1995 ? S 0:00 [socknal_cd00] 1996 ? S 0:00 [socknal_cd01] 1997 ? S 0:00 [socknal_cd02] 1998 ? S 0:00 [socknal_cd03] 1999 ? S 0:00 [socknal_reaper] 2000 ? S 0:00 [socknal_sd00_00] 2001 ? S 0:00 [socknal_sd00_01] 2002 ? S 0:19 [socknal_sd01_00] 2003 ? S 0:19 [socknal_sd01_01] 2004 ? S 0:00 [acceptor_001] 2005 ? S 0:00 [monitor_thread] 2006 ? S 0:00 [lnet_discovery] 2007 ? S 0:00 [ptlrpcd_rcv] 2008 ? S 0:06 [ptlrpcd_00_00] 2009 ? S 0:06 [ptlrpcd_00_01] 2010 ? S 0:03 [ptlrpcd_01_00] 2011 ? S 0:03 [ptlrpcd_01_01] 2390 ? S< 0:00 [kworker/1:1H] 3263 ? S 0:00 /usr/lib/systemd/systemd-udevd 3285 pts/0 R+ 0:00 ps ax 7661 pts/0 S+ 0:01 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 7808 ? S 0:00 [kworker/u8:3] 8575 ? S 0:00 [ldlm_cb00_000] 8576 ? S 0:00 [ldlm_cb00_001] 8577 ? S 0:01 [ldlm_cb01_000] 8578 ? S 0:01 [ldlm_cb01_001] 8579 ? S 0:00 [ldlm_cn00_000] 8580 ? S 0:00 [ldlm_cn00_001] 8581 ? S 0:00 [ldlm_cn00_002] 8582 ? S 0:00 [ldlm_cn01_000] 8583 ? S 0:00 [ldlm_cn01_001] 8584 ? S 0:00 [ldlm_cn01_002] 8585 ? S 0:01 [ldlm_bl_01] 8586 ? S 0:01 [ldlm_bl_02] 8587 ? S 0:00 [ldlm_elt] 8588 ? S 0:00 [ll_cfg_requeue] 16365 ? S 0:01 [ldlm_bl_03] 16367 ? S 0:02 [ldlm_cb01_002] 23734 ? S 0:00 [kworker/3:0] 24087 ? S 0:00 [kworker/3:2] 24259 ? S 0:00 [kworker/0:2] 25593 ? S 0:00 [kworker/1:2] 26090 ? S 0:00 [kworker/1:0] 26237 ? S 0:00 [kworker/0:0] 26342 ? S 0:00 [ldlm_bl_04] 26343 ? S 0:00 [ldlm_bl_05] 27108 ? S 0:02 [kworker/2:2] 29592 ? S 0:00 [kworker/2:0] 30501 ? S 0:00 [ldlm_bl_06] 30523 ? S 0:00 [ldlm_bl_07] 30657 ? S 0:00 [ldlm_cb00_002] 30658 ? S 0:00 [ldlm_bl_08] 30659 ? S 0:00 [ldlm_bl_09] sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746143855.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg203-server.1746143855.log.SNgfd9" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg203-server.1746143855.log.1pSQhV" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]