== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:03:07 (1746129787) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5113696 .. done Extending file size to 549280 .. Extending file size to 3122944 .. Extending file size to 7952864 .. Extending file size to 8588032 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9147744 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9207712 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9308800 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 2057 2059 2060 2061 2062 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) 192.168.202.145@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 16384-9293824 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 16384-9293824 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:05 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:01 [kworker/u8:0] 6 ? S 0:01 [ksoftirqd/0] 7 ? S 0:04 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:28 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:11 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:03 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:11 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:01 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:09 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:02 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:09 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 46 ? S 0:00 [kworker/0:1] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 51 ? S 0:00 [kworker/3:1] 53 ? S 0:00 [kswapd0] 54 ? SN 0:00 [ksmd] 55 ? SN 0:00 [khugepaged] 56 ? S< 0:00 [nfsiod] 57 ? S< 0:00 [crypto] 65 ? S< 0:00 [kthrotld] 67 ? S< 0:00 [kmpath_rdacd] 68 ? S< 0:00 [kpsmoused] 69 ? S< 0:00 [ipv6_addrconf] 82 ? S< 0:00 [deferwq] 277 ? S< 0:00 [ata_sff] 280 ? S 0:00 [scsi_eh_0] 281 ? S< 0:00 [scsi_tmf_0] 282 ? S 0:00 [scsi_eh_1] 283 ? S< 0:00 [scsi_tmf_1] 289 ? S 0:00 [kworker/u8:2] 291 ? S 0:00 [kworker/1:2] 449 ? Ss 0:01 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 451 ? S< 0:01 [nbd0] 479 ? S 0:00 [jbd2/nbd0-8] 480 ? S< 0:00 [ext4-rsv-conver] 536 ? S 0:00 [ldlm_bl_08] 564 ? Ss 0:02 /usr/lib/systemd/systemd-journald 598 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 712 ? S< 0:00 [kworker/0:1H] 714 ? S< 0:00 [kworker/3:1H] 723 ? S< 0:00 [kvm-irqfd-clean] 727 ? S< 0:00 [edac-poller] 769 ? S< 0:00 [kworker/1:1H] 770 ? S 0:00 [nfsv4.1-svc] 772 ? S< 0:00 [kworker/2:1H] 797 ? S 0:03 [kworker/2:2] 806 ? Ss 0:01 /usr/lib/systemd/systemd-logind 807 ? Ss 0:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 810 ? Ssl 0:00 /usr/sbin/gssproxy -D 818 ? Ssl 0:00 /usr/sbin/NetworkManager --no-daemon 827 ? Ss 0:00 /usr/sbin/sshd -D 842 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 844 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 846 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 861 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 865 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-0a143871-d977-4b12-83d2-e0a8483c6217-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 891 ? Ssl 0:01 /usr/sbin/rsyslogd -n 1119 ? Ss 0:00 sshd: root@pts/0 1163 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1791 ? S 0:00 [kworker/u8:3] 1803 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 1804 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg245-client.log 1996 ? S 0:00 [ptlrpc_hr00_000] 1997 ? S 0:00 [ptlrpc_hr01_000] 1998 ? S 0:00 [socknal_cd00] 1999 ? S 0:00 [socknal_cd01] 2000 ? S 0:00 [socknal_cd02] 2001 ? S 0:00 [socknal_cd03] 2002 ? S 0:00 [socknal_reaper] 2003 ? S 0:00 [socknal_sd00_00] 2004 ? S 0:00 [socknal_sd00_01] 2005 ? S 0:19 [socknal_sd01_00] 2006 ? S 0:19 [socknal_sd01_01] 2007 ? S 0:00 [acceptor_001] 2008 ? S 0:00 [monitor_thread] 2009 ? S 0:00 [lnet_discovery] 2010 ? S 0:00 [ptlrpcd_rcv] 2011 ? S 0:06 [ptlrpcd_00_00] 2012 ? S 0:06 [ptlrpcd_00_01] 2013 ? S 0:03 [ptlrpcd_01_00] 2014 ? S 0:03 [ptlrpcd_01_01] 4564 ? S 0:00 /usr/lib/systemd/systemd-udevd 4586 pts/0 R+ 0:00 ps ax 6802 ? S 0:00 [kworker/3:2] 7661 pts/0 S+ 0:01 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 8481 ? S 0:00 [ldlm_cb00_000] 8482 ? S 0:00 [ldlm_cb00_001] 8483 ? S 0:02 [ldlm_cb01_000] 8484 ? S 0:02 [ldlm_cb01_001] 8485 ? S 0:00 [ldlm_cn00_000] 8486 ? S 0:00 [ldlm_cn00_001] 8487 ? S 0:00 [ldlm_cn00_002] 8488 ? S 0:00 [ldlm_cn01_000] 8489 ? S 0:00 [ldlm_cn01_001] 8490 ? S 0:00 [ldlm_cn01_002] 8491 ? S 0:01 [ldlm_bl_01] 8492 ? S 0:01 [ldlm_bl_02] 8493 ? S 0:00 [ldlm_elt] 8494 ? S 0:00 [ll_cfg_requeue] 13498 ? S 0:02 [ldlm_cb01_002] 15609 ? S 0:03 [kworker/3:0] 16268 ? S 0:01 [ldlm_bl_03] 26239 ? S 0:01 [ldlm_bl_04] 26773 ? S 0:00 [ldlm_cb00_002] 27387 ? S 0:01 [kworker/2:1] 28214 ? S 0:00 [kworker/0:0] 28306 ? S 0:00 [kworker/u8:1] 29078 ? S 0:00 [kworker/2:0] 29244 ? S 0:00 [kworker/1:1] 29985 ? S 0:00 [kworker/1:0] 31162 ? S 0:00 [ldlm_bl_05] 31163 ? S 0:00 [ldlm_bl_06] 31524 ? S 0:00 [ldlm_bl_07] sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746129850.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg245-server.1746129850.log.JvlNou" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg245-server.1746129850.log.WFVg4W" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]