== sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:46:28 (1746135988) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre2 Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3986688 .. Extending file size to 2593184 .. done Extending file size to 5950048 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6632960 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8005056 .. Extending file size to 7813056 .. Extending file size to 8489088 .. Extending file size to 8880032 .. failed Extending file size to 8139616 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8152384 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8493696 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8773056 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9241568 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 28970 28971 28975 28976 28977 192.168.201.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre2 (opts:) 192.168.201.157@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 0-9244672 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-9244672 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:07 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:01 [kworker/u8:0] 6 ? S 0:02 [ksoftirqd/0] 7 ? S 0:05 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:40 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:13 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:04 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:12 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:03 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:13 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:04 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:13 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 49 ? S 0:00 [kworker/2:1] 52 ? S 0:00 [kswapd0] 53 ? SN 0:00 [ksmd] 54 ? SN 0:00 [khugepaged] 55 ? S< 0:00 [nfsiod] 56 ? S< 0:00 [crypto] 64 ? S< 0:00 [kthrotld] 66 ? S< 0:00 [kmpath_rdacd] 67 ? S< 0:00 [kpsmoused] 68 ? S< 0:00 [ipv6_addrconf] 69 ? S 0:03 [kworker/1:1] 82 ? S< 0:00 [deferwq] 287 ? S< 0:00 [ata_sff] 295 ? S 0:00 [scsi_eh_0] 296 ? S< 0:00 [scsi_tmf_0] 297 ? S 0:00 [scsi_eh_1] 298 ? S< 0:00 [scsi_tmf_1] 299 ? S 0:02 [kworker/u8:2] 450 ? Ss 0:01 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 452 ? S< 0:00 [nbd0] 454 ? S< 0:00 [kworker/0:1H] 455 ? S< 0:00 [kworker/2:1H] 483 ? S 0:00 [jbd2/nbd0-8] 484 ? S< 0:00 [ext4-rsv-conver] 568 ? Ss 0:02 /usr/lib/systemd/systemd-journald 583 ? S 0:00 [kworker/3:2] 601 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 716 ? S< 0:00 [kvm-irqfd-clean] 721 ? S< 0:00 [edac-poller] 772 ? S< 0:00 [kworker/1:1H] 773 ? S 0:00 [nfsv4.1-svc] 808 ? Ss 0:02 /usr/lib/systemd/systemd-logind 809 ? Ss 0:04 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 812 ? Ssl 0:00 /usr/sbin/gssproxy -D 820 ? Ssl 0:01 /usr/sbin/NetworkManager --no-daemon 826 ? Ss 0:00 /usr/sbin/sshd -D 846 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 847 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 848 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 862 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 865 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-bf0478f0-385c-468a-bdf2-b3ce04ecf134-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 896 ? Ssl 0:02 /usr/sbin/rsyslogd -n 999 ? S< 0:00 [kworker/3:1H] 1127 ? Ss 0:00 sshd: root@pts/0 1166 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1948 ? S 0:00 [ptlrpc_hr00_000] 1949 ? S 0:00 [ptlrpc_hr01_000] 1950 ? S 0:00 [socknal_cd00] 1951 ? S 0:00 [socknal_cd01] 1952 ? S 0:00 [socknal_cd02] 1953 ? S 0:00 [socknal_cd03] 1954 ? S 0:00 [socknal_reaper] 1955 ? S 0:15 [socknal_sd00_00] 1956 ? S 0:15 [socknal_sd00_01] 1957 ? S 0:00 [socknal_sd01_00] 1958 ? S 0:00 [socknal_sd01_01] 1959 ? S 0:00 [acceptor_001] 1960 ? S 0:00 [monitor_thread] 1961 ? S 0:00 [lnet_discovery] 1962 ? S 0:00 [ptlrpcd_rcv] 1963 ? S 0:03 [ptlrpcd_00_00] 1964 ? S 0:03 [ptlrpcd_00_01] 1965 ? S 0:05 [ptlrpcd_01_00] 1966 ? S 0:06 [ptlrpcd_01_01] 7037 pts/0 S+ 0:02 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 7718 ? S 0:00 [kworker/u8:3] 7755 ? S 0:01 [ldlm_cb00_000] 7756 ? S 0:01 [ldlm_cb00_001] 7757 ? S 0:00 [ldlm_cb01_000] 7758 ? S 0:00 [ldlm_cb01_001] 7759 ? S 0:00 [ldlm_cn00_000] 7760 ? S 0:00 [ldlm_cn00_001] 7761 ? S 0:00 [ldlm_cn00_002] 7762 ? S 0:00 [ldlm_cn01_000] 7763 ? S 0:00 [ldlm_cn01_001] 7764 ? S 0:00 [ldlm_cn01_002] 7765 ? S 0:01 [ldlm_bl_01] 7766 ? S 0:00 [ldlm_bl_02] 7767 ? S 0:00 [ldlm_elt] 7768 ? S 0:00 [ll_cfg_requeue] 15491 ? S 0:00 [ldlm_bl_03] 17613 ? S 0:00 [kworker/1:2] 18259 ? S 0:00 [kworker/0:2] 19734 ? S 0:01 [ldlm_cb00_002] 19735 ? S 0:01 [ldlm_bl_04] 20683 ? S 0:00 [kworker/3:0] 23872 ? S 0:01 [kworker/0:0] 26898 ? S 0:00 [ldlm_bl_05] 26899 ? S 0:00 [ldlm_bl_06] 26900 ? S 0:00 [ldlm_bl_07] 26901 ? S 0:00 [ldlm_bl_08] 28740 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 28741 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg157-client.log 28856 ? S 0:00 [kworker/2:0] 29188 ? S 0:00 [ldlm_cb01_002] 29501 ? S 0:00 [ldlm_bl_09] 30389 ? S 0:00 /usr/lib/systemd/systemd-udevd 30411 pts/0 R+ 0:00 ps ax sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746136054.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg157-server.1746136054.log.29BXhe" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg157-server.1746136054.log.OnF6MD" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]