== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:26:57 (1746156417) Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre2 Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 922048 .. Extending file size to 6141888 .. Extending file size to 8800064 .. Extending file size to 5241760 .. Extending file size to 7796064 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7874656 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9257280 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8379488 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8757280 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8780832 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9271296 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8857152 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9013888 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 910 912 915 916 917 192.168.201.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg122-client.virtnet /mnt/lustre2 (opts:) 192.168.201.122@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg122-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 16384-9256960 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 16384-9256960 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:07 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:01 [kworker/u8:0] 6 ? S 0:02 [ksoftirqd/0] 7 ? S 0:05 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:40 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:14 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:04 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 17 ? S 0:00 [kworker/1:0] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:14 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:02 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:12 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:03 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:12 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 46 ? S 0:00 [kworker/0:1] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 52 ? S 0:00 [kswapd0] 53 ? SN 0:00 [ksmd] 54 ? SN 0:00 [khugepaged] 55 ? S< 0:00 [nfsiod] 56 ? S< 0:00 [crypto] 64 ? S< 0:00 [kthrotld] 65 ? S 0:02 [kworker/u8:1] 66 ? S< 0:00 [kmpath_rdacd] 67 ? S< 0:00 [kpsmoused] 69 ? S< 0:00 [ipv6_addrconf] 82 ? S< 0:00 [deferwq] 276 ? S< 0:00 [ata_sff] 278 ? S 0:00 [scsi_eh_0] 279 ? S< 0:00 [scsi_tmf_0] 281 ? S 0:00 [scsi_eh_1] 282 ? S< 0:00 [scsi_tmf_1] 449 ? Ss 0:01 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 451 ? S< 0:00 [nbd0] 480 ? S 0:00 [jbd2/nbd0-8] 481 ? S< 0:00 [ext4-rsv-conver] 542 ? S< 0:00 [kworker/2:1H] 543 ? S< 0:00 [kworker/0:1H] 568 ? Ss 0:02 /usr/lib/systemd/systemd-journald 598 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 633 ? S 0:00 [kworker/u8:3] 645 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 646 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg122-client.log 713 ? S< 0:00 [kvm-irqfd-clean] 715 ? S< 0:00 [edac-poller] 769 ? S< 0:00 [kworker/3:1H] 770 ? S 0:00 [nfsv4.1-svc] 804 ? Ss 0:04 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 809 ? Ssl 0:01 /usr/sbin/NetworkManager --no-daemon 810 ? Ss 0:02 /usr/lib/systemd/systemd-logind 813 ? Ssl 0:00 /usr/sbin/gssproxy -D 826 ? Ss 0:00 /usr/sbin/sshd -D 843 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 844 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 845 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 859 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 864 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-3126061b-1b27-495c-8198-6e249ea0e5e4-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 891 ? Ssl 0:02 /usr/sbin/rsyslogd -n 1069 ? S< 0:00 [kworker/1:1H] 1133 ? Ss 0:00 sshd: root@pts/0 1172 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 2005 ? S 0:00 [ptlrpc_hr00_000] 2006 ? S 0:00 [ptlrpc_hr01_000] 2007 ? S 0:00 [socknal_cd00] 2008 ? S 0:00 [socknal_cd01] 2009 ? S 0:00 [socknal_cd02] 2010 ? S 0:00 [socknal_cd03] 2011 ? S 0:00 [socknal_reaper] 2012 ? S 0:00 [socknal_sd00_00] 2013 ? S 0:00 [socknal_sd00_01] 2014 ? S 0:19 [socknal_sd01_00] 2015 ? S 0:19 [socknal_sd01_01] 2016 ? S 0:00 [acceptor_001] 2017 ? S 0:00 [monitor_thread] 2018 ? S 0:00 [lnet_discovery] 2019 ? S 0:00 [ptlrpcd_rcv] 2020 ? S 0:06 [ptlrpcd_00_00] 2021 ? S 0:06 [ptlrpcd_00_01] 2022 ? S 0:03 [ptlrpcd_01_00] 2023 ? S 0:03 [ptlrpcd_01_01] 2686 ? S 0:00 /usr/lib/systemd/systemd-udevd 2708 pts/0 R+ 0:00 ps ax 7723 pts/0 S+ 0:02 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 8607 ? S 0:00 [kworker/u8:2] 8643 ? S 0:00 [ldlm_cb00_000] 8644 ? S 0:00 [ldlm_cb00_001] 8645 ? S 0:01 [ldlm_cb01_000] 8646 ? S 0:02 [ldlm_cb01_001] 8647 ? S 0:00 [ldlm_cn00_000] 8648 ? S 0:00 [ldlm_cn00_001] 8649 ? S 0:00 [ldlm_cn00_002] 8650 ? S 0:00 [ldlm_cn01_000] 8651 ? S 0:00 [ldlm_cn01_001] 8652 ? S 0:00 [ldlm_cn01_002] 8653 ? S 0:01 [ldlm_bl_01] 8654 ? S 0:01 [ldlm_bl_02] 8655 ? S 0:00 [ldlm_elt] 8656 ? S 0:00 [ll_cfg_requeue] 12688 ? S 0:00 [kworker/3:1] 16259 ? S 0:00 [kworker/2:1] 16441 ? S 0:01 [ldlm_bl_03] 16470 ? S 0:01 [ldlm_cb01_002] 17135 ? S 0:00 [kworker/1:2] 17356 ? S 0:03 [kworker/3:0] 18670 ? S 0:03 [kworker/2:0] 19620 ? S 0:00 [kworker/0:2] 26473 ? S 0:01 [ldlm_bl_04] 27047 ? S 0:00 [ldlm_cb00_002] 30723 ? S 0:00 [ldlm_bl_05] 30724 ? S 0:00 [ldlm_bl_06] 30903 ? S 0:00 [ldlm_bl_07] 31580 ? S 0:00 [ldlm_bl_08] 32158 ? S 0:00 [ldlm_bl_09] sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746156482.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg122-server.1746156482.log.ZMktJF" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg122-server.1746156482.log.XvSMSw" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]