-----============= acceptance-small: sanity-flr ============----- Wed Aug 12 15:54:45 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg157-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:55:05 (1786564505) === oleg157-client.virtnet: executing check_config_client /mnt/lustre oleg157-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg157-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff89fbc8417800.idle_timeout=debug osc.lustre-OST0001-osc-ffff89fbc8417800.idle_timeout=debug disable quota as required oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:55:41 (1786564541) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7627] == sanity-flr test 0a: lfs mirror create with -N option == 15:55:44 (1786564544) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:56:02 (1786564562) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:56:07 (1786564567) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:56:11 (1786564571) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:56:32 (1786564592) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:56:36 (1786564596) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:56:40 (1786564600) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff89fbc8417800.stats=clear osc.lustre-OST0001-osc-ffff89fbc8417800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:56:50 (1786564610) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (15s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:57:05 (1786564625) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:57:13 (1786564633) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:57:16 (1786564636) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created error: lfs extend: layout is NULL /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x16 lmm_fid: [0x200000401:0x16:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x20:0x0] } - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x16 lmm_fid: [0x200000401:0x16:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x20:0x0] } sanity-flr test_0m: @@@@@@ FAIL: failed to call extend /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:899:test_0m() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:909:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0m.*.1786564641.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0m.debug_log.oleg157-server.1786564641.log.yGZrEQ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0m.dmesg.oleg157-server.1786564641.log.Gi1N8d" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 0m (11s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:57:35 (1786564655) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x18:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 15:57:44 (1786564664) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:57:51 (1786564671) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:57:54 (1786564674) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory error: lfs extend: layout is NULL PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:58:03 (1786564683) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:58:15 (1786564695) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:58:22 (1786564702) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 2.09871 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100067 s, 10.5 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:58:34 (1786564714) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.121424 s, 25.9 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:58:42 (1786564722) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:58:54 (1786564734) Stopping /mnt/lustre-ost1 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89fbc8417800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89fbc8417800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg157-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89fbc8417800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89fbc8417800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg157-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (72s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:00:06 (1786564806) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.612178 s, 18.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89fbc8417800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89fbc8417800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg157-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:00:54 (1786564854) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89fbc8417800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89fbc8417800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg157-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89fbc8417800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89fbc8417800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg157-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:02:16 (1786564936) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.44586 s, 11.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 12 16:02 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.184s user 0m0.000s sys 0m0.177s Stopping /mnt/lustre-ost1 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89fbc8417800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89fbc8417800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.005s user 0m0.001s sys 0m0.187s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg157-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89fbc8417800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89fbc8417800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.199s user 0m0.001s sys 0m0.167s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg157-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg157-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89fbc8417800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89fbc8417800.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:03:46 (1786565026) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:03:50 (1786565030) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:03:53 (1786565033) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 16:03:57 (1786565037) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 16:04:04 (1786565044) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.591288 s, 21.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:04:17 (1786565057) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237221 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258436 s, 40.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:04:25 (1786565065) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:04:33 (1786565073) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0012061 s, 849 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00213544 s, 480 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00100462 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00187725 s, 545 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 16:04:43 (1786565083) create test files with size 15735303 .. 2e585b1ae49c14feaefd99586df26364 - 949f5dc368272745c2a55a93df3444bd - 5db4a98102c15e2f01624124331b2635 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg157-client.virtnet /mnt/lustre (opts:) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 16:05:07 (1786565107) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:05:26 (1786565126) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:05:33 (1786565133) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125429 s, 8.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602821 s, 17.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 16:05:42 (1786565142) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12645 s, 16.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.27642 s, 15.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174408 s, 12.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 16:05:57 (1786565157) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 16:06:01 (1786565161) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:06:04 (1786565164) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309634 s, 33.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 16:06:13 (1786565173) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:06:16 (1786565176) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } split mirror_id 1 id 0x10001 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } split mirror_id 2 id 0x20001 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:06:27 (1786565187) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.294002 s, 35.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg157-client.virtnet /mnt/lustre (opts:) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:06:38 (1786565198) llite.lustre-ffff89fbe0f90800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.363851 s, 28.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg157-client.virtnet /mnt/lustre (opts:) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff89fbd800e800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:06:55 (1786565215) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (48s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:07:43 (1786565263) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (47s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:08:30 (1786565310) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 16:08:35 (1786565315) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.280728 s, 11.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.309378 s, 10.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x43840945 (nosync,stale) Mirror 2: 0x43840945 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2a547132fc39af25abb8cccae32149a7 - mirror 1 checksum: 7f2f5b395c0eba7197218349a54cca67 - mirror 2 checksum: 2a547132fc39af25abb8cccae32149a7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T16:08:39-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T16:08:39-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2a547132fc39af25abb8cccae32149a7 - mirror 1 checksum: 7f2f5b395c0eba7197218349a54cca67 - mirror 2 checksum: 7f2f5b395c0eba7197218349a54cca67 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T16:08:39-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T16:08:39-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (17s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:08:54 (1786565334) Failing mds1 on oleg157-server Stopping /mnt/lustre-mds1 (opts:) on oleg157-server 16:09:02 (1786565342) shut down facet: mds1 facet_host: oleg157-server facet_failover_host: oleg157-server Failover mds1 to oleg157-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0000 16:09:28 (1786565368) targets are mounted 16:09:28 (1786565368) facet_failover done oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198039 s, 52.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (52s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:09:46 (1786565386) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:09:50 (1786565390) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:09:54 (1786565394) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0105528 s, 1.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:10:02 (1786565402) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:10:06 (1786565406) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00532224 s, 770 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0782152 s, 52.4 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:10:15 (1786565415) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:10:39 (1786565439) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:11:07 (1786565467) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:11:30 (1786565490) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:11:54 (1786565514) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 16:12:02 (1786565522) mirror create pid 54473 mirror split pid 54474 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54473 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54474 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 16:13:09 (1786565589) Seed set to 5591 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1786565591.573258 write 0x014d3d thru 0x0169f5 (0x1cb9 bytes) 000005 1786565591.575378 write 0x008a97 thru 0x00decc (0x5436 bytes) 000006 1786565591.591373 mapread 0x00ce56 thru 0x0169f5 (0x9ba0 bytes) 000007 1786565591.606946 mapread 0x00dc41 thru 0x00eda5 (0x1165 bytes) 000008 1786565591.607941 read 0x007f54 thru 0x00feb6 (0x7f63 bytes) 000011 1786565591.621122 MIRROR_EXTEND to 2 mirrors 000014 1786565592.341045 mapwrite 0x01e3fa thru 0x0219bc (0x35c3 bytes) 000016 1786565592.480974 read 0x00f8bc thru 0x012f7d (0x36c2 bytes) truncating to largest ever: 0x23fc5 000020 1786565592.489913 trunc from 0x0219bd to 0x023fc4 (0x2608 bytes) 000024 1786565592.511561 trunc from 0x023fc5 to 0x0154e9 (0xeadb bytes) 000025 1786565592.535543 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000026 1786565592.667067 MIRROR_EXTEND to 2 mirrors 000028 1786565593.417293 read 0x007894 thru 0x012083 (0xa7f0 bytes) 000029 1786565593.441976 read 0x0048ee thru 0x00d368 (0x8a7b bytes) 000030 1786565593.456274 mapread 0x00653e thru 0x007e64 (0x1927 bytes) 000034 1786565593.493117 write 0x03e93a thru 0x03ffff (0x16c6 bytes) 000039 1786565593.554881 mapread 0x018618 thru 0x01f2fc (0x6ce5 bytes) 000041 1786565593.563719 write 0x037248 thru 0x03ffff (0x8db8 bytes) truncating to largest ever: 0x3a7f9 000042 1786565593.566529 trunc from 0x040000 to 0x03a7f8 (0x5807 bytes) 000044 1786565593.590231 read 0x01f26b thru 0x01fd0e (0xaa4 bytes) 000046 1786565593.592996 mapwrite 0x02b649 thru 0x031cf1 (0x66a9 bytes) 000047 1786565593.649219 mapread 0x02a891 thru 0x039c32 (0xf3a2 bytes) 000048 1786565593.650690 read 0x036812 thru 0x03a7f8 (0x3fe7 bytes) 000050 1786565593.653938 trunc from 0x03a7f9 to 0x03a67a (0x17e bytes) 000051 1786565593.679028 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000054 1786565593.813421 mapread 0x0282d1 thru 0x028423 (0x153 bytes) 000055 1786565593.821460 trunc from 0x03a67b to 0x00b1e4 (0x2f496 bytes) 000057 1786565593.844460 trunc from 0x00b1e5 to 0x008639 (0x2bab bytes) 000058 1786565593.861154 mapread 0x0084ce thru 0x008639 (0x16c bytes) 000059 1786565593.867361 read 0x00802a thru 0x008639 (0x610 bytes) 000063 1786565593.871449 mapwrite 0x014343 thru 0x01ac4d (0x690b bytes) 000064 1786565594.005552 mapread 0x005ac1 thru 0x00c1fe (0x673e bytes) 000065 1786565594.024892 mapread 0x011da3 thru 0x018bfb (0x6e59 bytes) 000068 1786565594.029249 mapwrite 0x008105 thru 0x014ba1 (0xca9d bytes) 000070 1786565594.107353 write 0x033988 thru 0x03ffff (0xc678 bytes) 000072 1786565594.116158 mapwrite 0x02c60d thru 0x037578 (0xaf6c bytes) 000073 1786565594.222174 write 0x0391f8 thru 0x03ffff (0x6e08 bytes) 000074 1786565594.223530 mapwrite 0x0378ac thru 0x03ffff (0x8754 bytes) 000075 1786565594.289526 mapwrite 0x014d9f thru 0x01c0e4 (0x7346 bytes) 000076 1786565594.384847 mapread 0x0179e0 thru 0x0263ea (0xea0b bytes) 000077 1786565594.410022 MIRROR_EXTEND to 2 mirrors 000078 1786565595.074519 write 0x034781 thru 0x03ffff (0xb87f bytes) 000079 1786565595.142505 mapwrite 0x004fa1 thru 0x00e379 (0x93d9 bytes) 000080 1786565595.226391 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000082 1786565595.378465 mapread 0x018858 thru 0x0252fd (0xcaa6 bytes) truncating to largest ever: 0x3c2bd 000084 1786565595.415055 trunc from 0x040000 to 0x03c2bc (0x3d43 bytes) 000085 1786565595.473203 read 0x020bbc thru 0x02a066 (0x94ab bytes) 000087 1786565595.477258 trunc from 0x03c2bd to 0x003129 (0x39193 bytes) 000088 1786565595.511301 trunc from 0x00312a to 0x00db01 (0xa9d8 bytes) 000089 1786565595.537031 trunc from 0x00db02 to 0x00ba4c (0x20b5 bytes) 000090 1786565595.555378 trunc from 0x00ba4d to 0x03322a (0x277de bytes) 000094 1786565595.580893 mapwrite 0x0320f3 thru 0x03bd5e (0x9c6c bytes) 000095 1786565595.668058 mapread 0x026e15 thru 0x033189 (0xc375 bytes) 000096 1786565595.683147 mapwrite 0x023bb5 thru 0x026539 (0x2985 bytes) 000097 1786565595.749217 trunc from 0x03bd5f to 0x03b61e (0x740 bytes) 000098 1786565595.773557 trunc from 0x03b61f to 0x02891f (0x12cff bytes) 000100 1786565595.816190 trunc from 0x028920 to 0x00d5d4 (0x1b34b bytes) 000101 1786565595.856973 trunc from 0x00d5d5 to 0x0328a7 (0x252d3 bytes) 000102 1786565595.881977 read 0x000b69 thru 0x00ca01 (0xbe99 bytes) 000104 1786565595.896228 mapread 0x013db2 thru 0x01bdec (0x803b bytes) 000105 1786565595.898646 trunc from 0x0328a8 to 0x018c2b (0x19c7c bytes) 000108 1786565595.927202 write 0x02313d thru 0x026b90 (0x3a54 bytes) 000109 1786565595.928684 mapwrite 0x03be2e thru 0x03e78e (0x2961 bytes) 000110 1786565596.057091 MIRROR_EXTEND to 2 mirrors 000112 1786565597.635796 mapread 0x022228 thru 0x030121 (0xdefa bytes) 000115 1786565597.723392 read 0x00ceaa thru 0x00d14d (0x2a4 bytes) 000117 1786565597.770031 read 0x019774 thru 0x01aac6 (0x1353 bytes) 000118 1786565597.805403 MIRROR_EXTEND to 3 mirrors 000119 1786565598.707277 MIRROR_RESYNC 3 mirrors 000120 1786565599.045296 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000123 1786565599.586306 mapread 0x03d1da thru 0x03d56c (0x393 bytes) 000125 1786565599.656322 mapread 0x005e67 thru 0x0127aa (0xc944 bytes) 000127 1786565599.715549 write 0x01d5ba thru 0x026135 (0x8b7c bytes) 000128 1786565599.802006 mapwrite 0x022a66 thru 0x030a64 (0xdfff bytes) 000132 1786565599.919338 mapwrite 0x0256b2 thru 0x03468f (0xefde bytes) 000133 1786565599.977465 write 0x034ef8 thru 0x03de02 (0x8f0b bytes) 000134 1786565599.984316 trunc from 0x03e78f to 0x014eec (0x298a2 bytes) 000136 1786565600.019248 mapwrite 0x0354f8 thru 0x03ffff (0xab08 bytes) 000137 1786565600.132305 mapwrite 0x01554f thru 0x019911 (0x43c3 bytes) 000138 1786565600.212760 mapwrite 0x03af22 thru 0x03ffff (0x50de bytes) 000141 1786565600.257659 write 0x033bca thru 0x03ffff (0xc436 bytes) 000142 1786565600.259285 mapread 0x018822 thru 0x01e0fb (0x58da bytes) 000145 1786565600.270791 read 0x0134ef thru 0x021082 (0xdb94 bytes) 000146 1786565600.276090 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3198b9af (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2f9e8240 Mirror 4: 0x2f9e8240 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000148 1786565601.295215 write 0x02327e thru 0x0281b1 (0x4f34 bytes) 000153 1786565601.524320 mapread 0x0285d4 thru 0x0358b6 (0xd2e3 bytes) 000154 1786565601.615002 mapread 0x01eb2c thru 0x026e3b (0x8310 bytes) 000156 1786565601.647029 mapread 0x032300 thru 0x037c36 (0x5937 bytes) 000158 1786565601.657446 write 0x036b8c thru 0x03ffff (0x9474 bytes) 000159 1786565601.659017 read 0x02d1d0 thru 0x02e4cb (0x12fc bytes) 000162 1786565601.668770 mapread 0x01a8ef thru 0x025452 (0xab64 bytes) 000163 1786565601.670083 mapwrite 0x031444 thru 0x03b856 (0xa413 bytes) 000165 1786565601.758371 mapread 0x034eff thru 0x03ffff (0xb101 bytes) 000166 1786565601.764777 trunc from 0x040000 to 0x01006b (0x2ff94 bytes) 000167 1786565601.850262 mapread 0x005acc thru 0x01006b (0xa5a0 bytes) 000168 1786565601.869223 mapwrite 0x03d606 thru 0x03ffff (0x29fa bytes) 000170 1786565601.997287 read 0x00b6ef thru 0x01a303 (0xec15 bytes) 000172 1786565602.019703 mapwrite 0x016dae thru 0x0249dc (0xdc2f bytes) 000173 1786565602.123025 read 0x021253 thru 0x02360d (0x23bb bytes) 000174 1786565602.124123 mapwrite 0x00f24c thru 0x017a34 (0x87e9 bytes) 000177 1786565602.176042 read 0x0178fc thru 0x01f481 (0x7b86 bytes) truncating to largest ever: 0x3c3b3 000178 1786565602.177244 trunc from 0x040000 to 0x03c3b2 (0x3c4d bytes) 000180 1786565602.196124 read 0x02aa42 thru 0x0346cc (0x9c8b bytes) 000181 1786565602.197681 mapwrite 0x01304d thru 0x016d57 (0x3d0b bytes) 000183 1786565602.240272 mapread 0x009204 thru 0x010f08 (0x7d05 bytes) 000184 1786565602.242396 write 0x019468 thru 0x01eb06 (0x569f bytes) 000185 1786565602.244673 mapwrite 0x022616 thru 0x025e75 (0x3860 bytes) 000186 1786565602.314864 mapwrite 0x033b9b thru 0x03bf73 (0x83d9 bytes) 000188 1786565602.360962 read 0x01767a thru 0x01debe (0x6845 bytes) 000189 1786565602.362225 mapwrite 0x02cbd5 thru 0x02da3d (0xe69 bytes) 000190 1786565602.415728 mapread 0x00fe05 thru 0x0176a1 (0x789d bytes) 000191 1786565602.437322 mapwrite 0x0200ac thru 0x029330 (0x9285 bytes) 000196 1786565602.504372 MIRROR_RESYNC 3 mirrors 000197 1786565603.387911 mapwrite 0x039ed3 thru 0x03ffff (0x612d bytes) 000198 1786565603.598212 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9e153281 (stale) Mirror 4: 0x9e153281 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x129dc756 Mirror 5: 0x129dc756 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000199 1786565604.839594 mapread 0x02bc01 thru 0x030d5a (0x515a bytes) 000201 1786565604.892695 trunc from 0x040000 to 0x03022e (0xfdd1 bytes) 000202 1786565604.979750 mapread 0x0118df thru 0x01f44e (0xdb70 bytes) 000205 1786565605.009757 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3022f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3022f): Mirror 3: 0xbfbc9d08 (stale) Mirror 4: 0xbfbc9d08 (stale) Mirror 5: 0xbfbc9d08 (stale) CRC-32 checksum value for chunk [0, 0x3022f): Mirror 1: 0xbfbc9d08 Mirror 6: 0xbfbc9d08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3022f: skipped 000206 1786565606.108260 write 0x020d66 thru 0x030a68 (0xfd03 bytes) 000207 1786565606.250295 read 0x005a1c thru 0x012dae (0xd393 bytes) 000210 1786565606.266303 trunc from 0x030a69 to 0x015681 (0x1b3e7 bytes) 000211 1786565606.294578 write 0x01752c thru 0x022a45 (0xb51a bytes) 000213 1786565606.305974 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x22a46): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22a46): Mirror 3: 0xe7f689fb (stale) Mirror 4: 0xe7f689fb (stale) Mirror 5: 0xe7f689fb (stale) Mirror 6: 0xe7f689fb (stale) CRC-32 checksum value for chunk [0, 0x22a46): Mirror 1: 0xed2823c0 Mirror 7: 0xed2823c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22a46: skipped 000215 1786565607.394182 trunc from 0x022a46 to 0x00aafc (0x17f49 bytes) 000218 1786565607.575038 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xaafd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaafd): Mirror 3: 0xed73e10e (stale) Mirror 4: 0xed73e10e (stale) Mirror 5: 0xed73e10e (stale) Mirror 6: 0xed73e10e (stale) Mirror 7: 0xed73e10e (stale) CRC-32 checksum value for chunk [0, 0xaafd): Mirror 1: 0xed73e10e Mirror 8: 0xed73e10e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaafd: skipped 000219 1786565608.495208 mapwrite 0x02c506 thru 0x03bbc0 (0xf6bb bytes) 000221 1786565608.647022 trunc from 0x03bbc1 to 0x00a710 (0x314b0 bytes) 000223 1786565608.672518 write 0x0296ee thru 0x03661c (0xcf2f bytes) 000224 1786565608.675511 read 0x0135c8 thru 0x0220b9 (0xeaf2 bytes) 000225 1786565608.682110 read 0x01bd22 thru 0x0294c4 (0xd7a3 bytes) 000226 1786565608.689809 mapwrite 0x01f3bd thru 0x026c4e (0x7892 bytes) 000228 1786565608.741191 MIRROR_RESYNC 7 mirrors 000230 1786565610.499364 trunc from 0x03661d to 0x00c942 (0x29cda bytes) 000231 1786565610.575079 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc943): Mirror 3: 0xbe044417 (stale) Mirror 4: 0xbe044417 (stale) Mirror 5: 0xbe044417 (stale) Mirror 7: 0xbe044417 (stale) Mirror 8: 0xbe044417 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc943): Mirror 3: 0xbe044417 (stale) Mirror 4: 0xbe044417 (stale) Mirror 5: 0xbe044417 (stale) Mirror 7: 0xbe044417 (stale) Mirror 8: 0xbe044417 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc943: skipped 000232 1786565610.894559 write 0x019cfe thru 0x01fe3d (0x6140 bytes) 000234 1786565610.897810 mapread 0x01c50f thru 0x01d8b6 (0x13a8 bytes) 000235 1786565610.899224 mapread 0x017bf7 thru 0x018881 (0xc8b bytes) 000237 1786565610.912493 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1fe3e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fe3e): Mirror 3: 0x8af20e84 (stale) Mirror 4: 0x8af20e84 (stale) Mirror 5: 0x8af20e84 (stale) Mirror 7: 0x8af20e84 (stale) Mirror 8: 0x8af20e84 (stale) CRC-32 checksum value for chunk [0, 0x1fe3e): Mirror 1: 0x8f225ba2 Mirror 9: 0x8f225ba2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fe3e: skipped 000238 1786565611.925057 mapwrite 0x02a243 thru 0x0347b6 (0xa574 bytes) 000240 1786565612.107356 read 0x02fd73 thru 0x0347b6 (0x4a44 bytes) 000241 1786565612.108110 mapread 0x0203b6 thru 0x020bd9 (0x824 bytes) 000242 1786565612.114619 trunc from 0x0347b7 to 0x020aeb (0x13ccb bytes) 000245 1786565612.151089 mapread 0x00b209 thru 0x01a0a0 (0xee98 bytes) 000246 1786565612.180136 mapread 0x01f853 thru 0x020aeb (0x1299 bytes) 000247 1786565612.181450 mapwrite 0x03d462 thru 0x03ffff (0x2b9e bytes) 000250 1786565612.256097 mapwrite 0x024fc4 thru 0x0272ad (0x22ea bytes) 000252 1786565612.320743 mapwrite 0x02af96 thru 0x03a986 (0xf9f1 bytes) 000254 1786565612.389788 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6b763930 (stale) Mirror 4: 0x6b763930 (stale) Mirror 5: 0x6b763930 (stale) Mirror 7: 0x6b763930 (stale) Mirror 9: 0xc17abbd3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6b763930 (stale) Mirror 4: 0x6b763930 (stale) Mirror 5: 0x6b763930 (stale) Mirror 7: 0x6b763930 (stale) Mirror 9: 0xc17abbd3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000255 1786565612.767288 trunc from 0x040000 to 0x0250a4 (0x1af5b bytes) 000256 1786565612.783968 mapwrite 0x029202 thru 0x02b87e (0x267d bytes) 000257 1786565612.902464 mapread 0x00ee2c thru 0x01ca5b (0xdc30 bytes) 000259 1786565612.931657 mapwrite 0x03c507 thru 0x03ffff (0x3af9 bytes) 000260 1786565613.026385 MIRROR_RESYNC 6 mirrors 000261 1786565615.617851 mapwrite 0x0110ad thru 0x017edc (0x6e30 bytes) 000264 1786565615.836468 trunc from 0x040000 to 0x0351fe (0xae01 bytes) 000267 1786565615.905005 mapwrite 0x00a6fc thru 0x00b8f3 (0x11f8 bytes) 000270 1786565616.016617 mapwrite 0x038ba7 thru 0x03ffff (0x7459 bytes) 000271 1786565616.159767 mapread 0x033f0b thru 0x03f8aa (0xb9a0 bytes) 000273 1786565616.163031 read 0x014757 thru 0x01c5ad (0x7e57 bytes) 000274 1786565616.172568 read 0x00ade7 thru 0x011ac5 (0x6cdf bytes) 000276 1786565616.178384 MIRROR_RESYNC 6 mirrors 000277 1786565618.352625 trunc from 0x040000 to 0x01634c (0x29cb3 bytes) 000278 1786565618.426205 trunc from 0x01634d to 0x00e7a5 (0x7ba7 bytes) 000279 1786565618.450246 mapread 0x004e2b thru 0x00b31e (0x64f4 bytes) 000280 1786565618.470928 read 0x00ab33 thru 0x00e7a5 (0x3c73 bytes) 000283 1786565618.473553 mapread 0x00222a thru 0x00488b (0x2662 bytes) 000284 1786565618.475522 trunc from 0x00e7a6 to 0x019856 (0xb0b1 bytes) 000285 1786565618.496945 write 0x01d593 thru 0x02b311 (0xdd7f bytes) 000286 1786565618.500089 read 0x01d3c0 thru 0x0208d9 (0x351a bytes) 000287 1786565618.505369 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b312): Mirror 3: 0x9dd43423 (stale) Mirror 4: 0x9dd43423 (stale) Mirror 7: 0x9dd43423 (stale) Mirror 9: 0x9dd43423 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b312): Mirror 3: 0x9dd43423 (stale) Mirror 4: 0x9dd43423 (stale) Mirror 7: 0x9dd43423 (stale) Mirror 9: 0x9dd43423 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b312: skipped 000289 1786565618.852968 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b312): Mirror 3: 0x9dd43423 (stale) Mirror 4: 0x9dd43423 (stale) Mirror 9: 0x9dd43423 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b312): Mirror 3: 0x9dd43423 (stale) Mirror 4: 0x9dd43423 (stale) Mirror 9: 0x9dd43423 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b312: skipped 000290 1786565619.141264 write 0x00df37 thru 0x01ca38 (0xeb02 bytes) 000291 1786565619.156776 read 0x000e4a thru 0x00dab3 (0xcc6a bytes) 000292 1786565619.170600 MIRROR_RESYNC 4 mirrors 000293 1786565619.982194 trunc from 0x02b312 to 0x015b7e (0x15793 bytes) 000296 1786565620.078773 read 0x008b42 thru 0x015b7e (0xd03d bytes) 000297 1786565620.093563 read 0x012d27 thru 0x013851 (0xb2b bytes) 000299 1786565620.095561 read 0x0091cd thru 0x015b7e (0xc9b2 bytes) 000301 1786565620.101627 MIRROR_RESYNC 4 mirrors 000303 1786565621.016187 read 0x00ad01 thru 0x015b7e (0xae7e bytes) 000304 1786565621.025055 read 0x00dd07 thru 0x015b7e (0x7e78 bytes) truncating to largest ever: 0x3cb5e 000306 1786565621.026460 trunc from 0x015b7f to 0x03cb5d (0x26fdf bytes) 000307 1786565621.103197 write 0x0028f7 thru 0x003269 (0x973 bytes) 000308 1786565621.120018 MIRROR_RESYNC 4 mirrors 000310 1786565622.485943 mapread 0x013f26 thru 0x01cbeb (0x8cc6 bytes) 000312 1786565622.502542 mapwrite 0x01dda7 thru 0x01ec28 (0xe82 bytes) 000314 1786565622.650813 trunc from 0x03cb5e to 0x030ba4 (0xbfb9 bytes) 000315 1786565622.668579 mapread 0x00515e thru 0x014d94 (0xfc37 bytes) 000316 1786565622.683815 mapwrite 0x014ade thru 0x01e53f (0x9a62 bytes) truncating to largest ever: 0x3d1fa 000319 1786565622.739345 trunc from 0x030ba5 to 0x03d1f9 (0xc655 bytes) 000320 1786565622.756674 read 0x01145c thru 0x019040 (0x7be5 bytes) 000322 1786565622.757969 mapwrite 0x03e663 thru 0x03fca1 (0x163f bytes) 000323 1786565622.828267 mapwrite 0x00b64a thru 0x0129d1 (0x7388 bytes) 000324 1786565622.947489 read 0x027b8a thru 0x0294b9 (0x1930 bytes) 000327 1786565622.958884 read 0x013da1 thru 0x017c33 (0x3e93 bytes) 000330 1786565622.962400 read 0x028c7a thru 0x0343b7 (0xb73e bytes) 000331 1786565622.968471 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fca2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fca2): Mirror 3: 0xcaf15cb (stale) Mirror 4: 0xcaf15cb (stale) Mirror 9: 0xcaf15cb (stale) CRC-32 checksum value for chunk [0, 0x3fca2): Mirror 1: 0x16a6f5b1 Mirror 10: 0x16a6f5b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fca2: skipped 000332 1786565624.583462 mapread 0x02a9f1 thru 0x039ef8 (0xf508 bytes) 000333 1786565624.633944 MIRROR_RESYNC 5 mirrors 000337 1786565626.420910 mapwrite 0x00cdb0 thru 0x0172cc (0xa51d bytes) 000340 1786565626.625307 MIRROR_RESYNC 5 mirrors 000341 1786565628.584691 read 0x02970c thru 0x038dcd (0xf6c2 bytes) 000343 1786565628.626592 write 0x002f8b thru 0x01107d (0xe0f3 bytes) 000345 1786565628.794503 mapread 0x036364 thru 0x03fca1 (0x993e bytes) 000348 1786565628.822559 mapwrite 0x03c2af thru 0x03ffff (0x3d51 bytes) 000349 1786565628.946616 mapread 0x016aa4 thru 0x01c8a3 (0x5e00 bytes) 000351 1786565628.972758 write 0x026df1 thru 0x02ff1c (0x912c bytes) 000352 1786565628.991787 read 0x027c81 thru 0x02c8b3 (0x4c33 bytes) 000353 1786565628.993208 mapwrite 0x02edaa thru 0x0345ed (0x5844 bytes) 000354 1786565629.086695 write 0x037aba thru 0x03836c (0x8b3 bytes) 000355 1786565629.089733 mapread 0x00f877 thru 0x01995c (0xa0e6 bytes) 000356 1786565629.094460 mapwrite 0x0085e9 thru 0x0137d7 (0xb1ef bytes) 000357 1786565629.207271 mapread 0x01beec thru 0x01f7d6 (0x38eb bytes) 000358 1786565629.209316 mapread 0x00349a thru 0x00f374 (0xbedb bytes) 000359 1786565629.217569 MIRROR_RESYNC 5 mirrors 000360 1786565630.506301 mapwrite 0x021d60 thru 0x0285c6 (0x6867 bytes) 000361 1786565630.664426 read 0x025fce thru 0x028462 (0x2495 bytes) 000362 1786565630.665786 read 0x02449b thru 0x033cc6 (0xf82c bytes) 000365 1786565630.683332 trunc from 0x040000 to 0x02ed50 (0x112af bytes) 000366 1786565630.708755 trunc from 0x02ed51 to 0x03060f (0x18bf bytes) 000367 1786565630.725328 write 0x009a78 thru 0x018b27 (0xf0b0 bytes) 000371 1786565630.732156 mapread 0x02cad1 thru 0x03060f (0x3b3f bytes) 000373 1786565630.745609 mapread 0x0167ac thru 0x01e1f0 (0x7a45 bytes) 000375 1786565630.748566 write 0x02aeba thru 0x02e90e (0x3a55 bytes) 000376 1786565630.749902 read 0x008494 thru 0x013fae (0xbb1b bytes) 000377 1786565630.751559 write 0x01e03a thru 0x02b5f5 (0xd5bc bytes) 000378 1786565630.753898 trunc from 0x030610 to 0x022a9c (0xdb73 bytes) 000379 1786565630.778995 mapwrite 0x0221bb thru 0x0250de (0x2f24 bytes) 000380 1786565630.865955 read 0x0068ac thru 0x00bfa5 (0x56fa bytes) 000382 1786565630.872990 mapwrite 0x019b0a thru 0x0204e5 (0x69dc bytes) 000383 1786565630.974924 write 0x012344 thru 0x021043 (0xed00 bytes) 000384 1786565630.976323 mapwrite 0x033ddd thru 0x03c4cc (0x86f0 bytes) 000386 1786565631.132605 write 0x00d245 thru 0x01904c (0xbe08 bytes) 000387 1786565631.138477 mapwrite 0x0293a9 thru 0x0344fc (0xb154 bytes) 000389 1786565631.215926 read 0x025121 thru 0x02e091 (0x8f71 bytes) 000390 1786565631.217588 trunc from 0x03c4cd to 0x01c372 (0x2015a bytes) 000391 1786565631.237123 write 0x02fcef thru 0x032305 (0x2617 bytes) 000393 1786565631.240027 mapread 0x0316e6 thru 0x032305 (0xc20 bytes) 000394 1786565631.242420 mapwrite 0x00b3f4 thru 0x00cad3 (0x16e0 bytes) 000395 1786565631.294995 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32306): Mirror 3: 0x47f690e5 (stale) Mirror 9: 0x47f690e5 (stale) Mirror 10: 0x47f690e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32306): Mirror 3: 0x47f690e5 (stale) Mirror 9: 0x47f690e5 (stale) Mirror 10: 0x47f690e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32306: skipped 000396 1786565631.672105 trunc from 0x032306 to 0x01090c (0x219f9 bytes) 000397 1786565631.694875 write 0x01aca5 thru 0x01f266 (0x45c2 bytes) 000399 1786565631.696599 trunc from 0x01f267 to 0x0354de (0x16278 bytes) 000401 1786565631.720622 mapread 0x00bc34 thru 0x01562b (0x99f8 bytes) 000402 1786565631.741397 trunc from 0x0354df to 0x0171a7 (0x1e337 bytes) 000404 1786565631.761453 trunc from 0x0171a8 to 0x00e8d8 (0x88cf bytes) 000405 1786565631.785143 mapread 0x00391e thru 0x00e8d8 (0xafbb bytes) 000406 1786565631.794705 read 0x009c00 thru 0x00e8d8 (0x4cd9 bytes) 000407 1786565631.796393 mapwrite 0x012c6e thru 0x022145 (0xf4d8 bytes) 000408 1786565631.886782 mapwrite 0x012c61 thru 0x0160e8 (0x3488 bytes) 000410 1786565631.939382 write 0x01f3a6 thru 0x0251dd (0x5e38 bytes) 000411 1786565631.941463 mapread 0x00cd5e thru 0x00fa5b (0x2cfe bytes) 000412 1786565631.943857 mapread 0x01e06d thru 0x020133 (0x20c7 bytes) 000414 1786565631.946204 read 0x01548e thru 0x0194dd (0x4050 bytes) 000415 1786565631.952105 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x251de): Mirror 3: 0x4d360c7c (stale) Mirror 10: 0x4d360c7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x251de): Mirror 3: 0x4d360c7c (stale) Mirror 10: 0x4d360c7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x251de: skipped 000420 1786565632.294704 MIRROR_RESYNC 3 mirrors 000422 1786565633.126176 mapread 0x010af8 thru 0x0155c5 (0x4ace bytes) 000427 1786565633.149506 read 0x001d3a thru 0x009377 (0x763e bytes) 000428 1786565633.159713 mapread 0x009487 thru 0x00ae3b (0x19b5 bytes) 000430 1786565633.165520 mapwrite 0x000091 thru 0x0024d9 (0x2449 bytes) 000431 1786565633.381469 read 0x019420 thru 0x021bfa (0x87db bytes) 000432 1786565633.398525 write 0x001156 thru 0x003b26 (0x29d1 bytes) 000433 1786565633.408430 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x251de): Mirror 3: 0xf3521bbd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x251de): Mirror 3: 0xf3521bbd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x251de: skipped 000435 1786565633.717158 read 0x0182ec thru 0x023df0 (0xbb05 bytes) 000436 1786565633.728190 read 0x021266 thru 0x0251dd (0x3f78 bytes) 000437 1786565633.741776 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x251de): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x251de): Mirror 3: 0xf3521bbd (stale) CRC-32 checksum value for chunk [0, 0x251de): Mirror 1: 0x1b11389e Mirror 4: 0x1b11389e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x251de: skipped 000439 1786565634.635353 read 0x021133 thru 0x0251dd (0x40ab bytes) 000444 1786565634.741236 write 0x00958c thru 0x0115d5 (0x804a bytes) 000445 1786565634.838784 trunc from 0x0251de to 0x007e87 (0x1d356 bytes) 000446 1786565634.862920 trunc from 0x007e88 to 0x016b40 (0xecb9 bytes) truncating to largest ever: 0x3fa95 000447 1786565634.876171 trunc from 0x016b41 to 0x03fa94 (0x28f54 bytes) 000449 1786565634.897016 read 0x02e644 thru 0x034de1 (0x679e bytes) 000450 1786565634.911960 mapwrite 0x0213bb thru 0x0276cb (0x6311 bytes) 000451 1786565634.980420 mapwrite 0x032b77 thru 0x03ca6f (0x9ef9 bytes) 000453 1786565635.056031 write 0x02a858 thru 0x02ef62 (0x470b bytes) 000454 1786565635.060611 write 0x00781b thru 0x00811a (0x900 bytes) 000458 1786565635.065587 trunc from 0x03fa95 to 0x0328b5 (0xd1df bytes) 000460 1786565635.105260 MIRROR_RESYNC 3 mirrors 000461 1786565635.912376 MIRROR_EXTEND to 4 mirrors 000464 1786565636.656552 mapread 0x022c05 thru 0x02b1d0 (0x85cc bytes) 000465 1786565636.685306 read 0x0301a0 thru 0x0328b5 (0x2716 bytes) 000466 1786565636.701164 MIRROR_EXTEND to 5 mirrors 000470 1786565637.228395 trunc from 0x0328b6 to 0x03609d (0x37e8 bytes) 000472 1786565637.284656 trunc from 0x03609e to 0x031308 (0x4d95 bytes) 000477 1786565637.299884 mapread 0x0140d3 thru 0x01f8dc (0xb80a bytes) 000478 1786565637.319960 read 0x023297 thru 0x031308 (0xe072 bytes) 000481 1786565637.330294 read 0x02440b thru 0x02e9d1 (0xa5c7 bytes) 000483 1786565637.331683 mapwrite 0x001e0f thru 0x0024cc (0x6be bytes) 000484 1786565637.379081 trunc from 0x031309 to 0x03b78f (0xa487 bytes) 000485 1786565637.392552 read 0x00425c thru 0x00dcec (0x9a91 bytes) 000486 1786565637.394192 mapwrite 0x00d2f4 thru 0x01b35b (0xe068 bytes) 000487 1786565637.473491 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b790): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b790): Mirror 3: 0x6ab9315e (stale) Mirror 4: 0x6ab9315e (stale) Mirror 5: 0x6ab9315e (stale) Mirror 6: 0x6ab9315e (stale) CRC-32 checksum value for chunk [0, 0x3b790): Mirror 1: 0xec819500 Mirror 7: 0xec819500 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b790: skipped 000490 1786565638.790712 mapwrite 0x02654b thru 0x02edf2 (0x88a8 bytes) 000492 1786565638.993582 mapwrite 0x0054de thru 0x00e12c (0x8c4f bytes) 000493 1786565639.084536 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b790): Mirror 3: 0x6ab9315e (stale) Mirror 4: 0x6ab9315e (stale) Mirror 5: 0x6ab9315e (stale) Mirror 7: 0xec819500 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b790): Mirror 3: 0x6ab9315e (stale) Mirror 4: 0x6ab9315e (stale) Mirror 5: 0x6ab9315e (stale) Mirror 7: 0xec819500 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b790: skipped 000496 1786565639.522301 mapread 0x03ae13 thru 0x03b78f (0x97d bytes) 000498 1786565639.541453 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b790): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b790): Mirror 3: 0x6ab9315e (stale) Mirror 4: 0x6ab9315e (stale) Mirror 5: 0x6ab9315e (stale) Mirror 7: 0xec819500 (stale) CRC-32 checksum value for chunk [0, 0x3b790): Mirror 1: 0x34732a0c Mirror 8: 0x34732a0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b790: skipped 000500 1786565641.344537 mapwrite 0x0291f3 thru 0x02f769 (0x6577 bytes) 000501 1786565641.603084 read 0x030649 thru 0x03a832 (0xa1ea bytes) 000502 1786565641.623251 write 0x01a60a thru 0x01fd41 (0x5738 bytes) 000503 1786565641.651159 mapwrite 0x00e062 thru 0x00ecf9 (0xc98 bytes) 000504 1786565641.798441 mapread 0x013c32 thru 0x01ca54 (0x8e23 bytes) 000506 1786565641.809099 mapread 0x03452b thru 0x03b78f (0x7265 bytes) 000507 1786565641.817425 mapread 0x006646 thru 0x00e0a4 (0x7a5f bytes) 000512 1786565641.832582 write 0x02a9c2 thru 0x02d067 (0x26a6 bytes) 000513 1786565641.841304 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b790): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b790): Mirror 3: 0x6ab9315e (stale) Mirror 4: 0x6ab9315e (stale) Mirror 5: 0x6ab9315e (stale) Mirror 7: 0xec819500 (stale) Mirror 8: 0x34732a0c (stale) CRC-32 checksum value for chunk [0, 0x3b790): Mirror 1: 0x1a27d534 Mirror 9: 0x1a27d534 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b790: skipped 000514 1786565643.312121 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b790): Mirror 3: 0x6ab9315e (stale) Mirror 4: 0x6ab9315e (stale) Mirror 5: 0x6ab9315e (stale) Mirror 8: 0x34732a0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b790): Mirror 3: 0x6ab9315e (stale) Mirror 4: 0x6ab9315e (stale) Mirror 5: 0x6ab9315e (stale) Mirror 8: 0x34732a0c (stale) CRC-32 checksum value for chunk [0, 0x3b790): Mirror 1: 0x1a27d534 Mirror 9: 0x1a27d534 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b790: skipped 000515 1786565644.114586 MIRROR_RESYNC 6 mirrors 000516 1786565645.278462 mapread 0x01c643 thru 0x01fd59 (0x3717 bytes) 000517 1786565645.296803 mapwrite 0x035fb8 thru 0x03ffff (0xa048 bytes) 000518 1786565645.440081 write 0x03a4cd thru 0x03ffff (0x5b33 bytes) 000522 1786565645.443340 read 0x02fe09 thru 0x03f415 (0xf60d bytes) 000524 1786565645.445206 mapwrite 0x02ace1 thru 0x03566c (0xa98c bytes) 000525 1786565645.492741 mapwrite 0x004eb9 thru 0x0129f5 (0xdb3d bytes) 000526 1786565645.565172 read 0x02c142 thru 0x03349e (0x735d bytes) 000527 1786565645.565748 read 0x015b38 thru 0x023e97 (0xe360 bytes) 000528 1786565645.574123 read 0x03c453 thru 0x03d05b (0xc09 bytes) 000529 1786565645.575086 mapread 0x02afd8 thru 0x03afaf (0xffd8 bytes) 000531 1786565645.578219 write 0x02bede thru 0x035a02 (0x9b25 bytes) 000533 1786565645.581996 mapread 0x02aae6 thru 0x032881 (0x7d9c bytes) 000535 1786565645.588982 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x143e2862 (stale) Mirror 4: 0x143e2862 (stale) Mirror 5: 0x143e2862 (stale) Mirror 8: 0x143e2862 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x143e2862 (stale) Mirror 4: 0x143e2862 (stale) Mirror 5: 0x143e2862 (stale) Mirror 8: 0x143e2862 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000537 1786565646.106448 trunc from 0x040000 to 0x029d9a (0x16265 bytes) 000539 1786565646.133422 mapread 0x001e8d thru 0x00f44f (0xd5c3 bytes) 000541 1786565646.167610 trunc from 0x029d9b to 0x038e72 (0xf0d8 bytes) 000542 1786565646.231307 read 0x020dc0 thru 0x0275ba (0x67fb bytes) 000545 1786565646.257452 mapread 0x02955e thru 0x02caff (0x35a2 bytes) 000548 1786565646.285080 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38e73): Mirror 3: 0xf5d60c9f (stale) Mirror 4: 0xf5d60c9f (stale) Mirror 5: 0xf5d60c9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38e73): Mirror 3: 0xf5d60c9f (stale) Mirror 4: 0xf5d60c9f (stale) Mirror 5: 0xf5d60c9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38e73: skipped 000551 1786565646.700066 mapread 0x016cf9 thru 0x016db5 (0xbd bytes) 000553 1786565646.712610 read 0x021359 thru 0x02ef54 (0xdbfc bytes) 000557 1786565646.728394 read 0x005462 thru 0x0126ea (0xd289 bytes) 000559 1786565646.740762 trunc from 0x038e73 to 0x007a11 (0x31461 bytes) 000560 1786565646.759989 mapwrite 0x02f264 thru 0x03b2b3 (0xc050 bytes) 000562 1786565646.881122 mapread 0x01583d thru 0x01604d (0x811 bytes) 000563 1786565646.899999 write 0x012404 thru 0x015a7c (0x3679 bytes) 000564 1786565646.909013 read 0x0113ee thru 0x016152 (0x4d65 bytes) 000565 1786565646.935461 write 0x00dcf7 thru 0x011f4f (0x4259 bytes) 000566 1786565646.942726 write 0x014e4a thru 0x01d9e5 (0x8b9c bytes) 000570 1786565646.953644 mapread 0x001202 thru 0x00730d (0x610c bytes) 000571 1786565646.961917 mapwrite 0x032baa thru 0x03db41 (0xaf98 bytes) 000574 1786565647.158981 write 0x01ec00 thru 0x02ad3a (0xc13b bytes) 000575 1786565647.171223 write 0x029e67 thru 0x036d7b (0xcf15 bytes) 000576 1786565647.172812 mapwrite 0x030594 thru 0x03dbce (0xd63b bytes) 000577 1786565647.306248 mapwrite 0x02b4dc thru 0x02c58e (0x10b3 bytes) 000578 1786565647.338595 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dbcf): Mirror 3: 0xfa8a3010 (stale) Mirror 4: 0xfa8a3010 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dbcf): Mirror 3: 0xfa8a3010 (stale) Mirror 4: 0xfa8a3010 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dbcf: skipped 000579 1786565647.640381 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dbcf): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dbcf): Mirror 3: 0xfa8a3010 (stale) Mirror 4: 0xfa8a3010 (stale) CRC-32 checksum value for chunk [0, 0x3dbcf): Mirror 1: 0x559b4bd Mirror 5: 0x559b4bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dbcf: skipped 000580 1786565648.592871 mapwrite 0x00d50a thru 0x01aaea (0xd5e1 bytes) 000581 1786565648.776453 mapread 0x02e3b3 thru 0x03185d (0x34ab bytes) 000587 1786565648.814137 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dbcf): Mirror 4: 0xfa8a3010 (stale) Mirror 5: 0x559b4bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dbcf): Mirror 4: 0xfa8a3010 (stale) Mirror 5: 0x559b4bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dbcf: skipped 000588 1786565649.212131 trunc from 0x03dbcf to 0x00baf7 (0x320d7 bytes) 000589 1786565649.227119 mapwrite 0x00bc5b thru 0x00fd5c (0x4102 bytes) 000590 1786565649.316300 mapwrite 0x02c8bb thru 0x02f956 (0x309c bytes) 000591 1786565649.391924 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f957): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f957): Mirror 4: 0x93190184 (stale) Mirror 5: 0xb279b892 (stale) CRC-32 checksum value for chunk [0, 0x2f957): Mirror 1: 0x74e96402 Mirror 6: 0x74e96402 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f957: skipped 000592 1786565650.381224 read 0x01ca94 thru 0x0284e6 (0xba53 bytes) 000593 1786565650.427676 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f957): Mirror 5: 0xb279b892 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f957): Mirror 5: 0xb279b892 (stale) CRC-32 checksum value for chunk [0, 0x2f957): Mirror 1: 0x74e96402 Mirror 6: 0x74e96402 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f957: skipped 000594 1786565651.509204 mapread 0x0252fa thru 0x02f956 (0xa65d bytes) 000595 1786565651.603921 mapwrite 0x018907 thru 0x0205fb (0x7cf5 bytes) 000597 1786565651.847239 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f957): Mirror 6: 0x74e96402 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f957): Mirror 6: 0x74e96402 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f957: skipped 000599 1786565652.164584 trunc from 0x02f957 to 0x007e10 (0x27b46 bytes) 000600 1786565652.204524 write 0x0017ee thru 0x00c2a5 (0xaab8 bytes) 000602 1786565652.212499 mapwrite 0x02f328 thru 0x03631e (0x6ff7 bytes) 000603 1786565652.300968 trunc from 0x03631f to 0x00724d (0x2f0d1 bytes) 000604 1786565652.322035 trunc from 0x00724e to 0x0300d0 (0x28e83 bytes) 000608 1786565652.342272 mapwrite 0x03efe4 thru 0x03ffff (0x101c bytes) 000611 1786565652.430901 trunc from 0x040000 to 0x022854 (0x1d7ab bytes) 000612 1786565652.454998 trunc from 0x022855 to 0x01baec (0x6d68 bytes) 000613 1786565652.473617 mapread 0x009c5e thru 0x012056 (0x83f9 bytes) 000615 1786565652.497520 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000616 1786565652.623312 write 0x011483 thru 0x01988e (0x840c bytes) 000617 1786565652.638278 write 0x036d1f thru 0x03e04a (0x732c bytes) 000620 1786565652.646209 write 0x00c8e1 thru 0x0137ee (0x6f0e bytes) 000621 1786565652.653020 write 0x00013f thru 0x000919 (0x7db bytes) 000622 1786565652.658100 trunc from 0x03e04b to 0x00d45a (0x30bf0 bytes) 000623 1786565652.674191 write 0x0131e1 thru 0x018865 (0x5685 bytes) 000624 1786565652.679567 MIRROR_EXTEND to 2 mirrors 000628 1786565653.896589 MIRROR_EXTEND to 3 mirrors 000629 1786565654.581158 mapread 0x014cd0 thru 0x018865 (0x3b96 bytes) 000630 1786565654.666723 MIRROR_RESYNC 3 mirrors 000632 1786565655.028551 mapwrite 0x030561 thru 0x035b6b (0x560b bytes) 000633 1786565655.187559 write 0x00e712 thru 0x018596 (0x9e85 bytes) 000640 1786565655.204482 trunc from 0x035b6c to 0x01d585 (0x185e6 bytes) 000644 1786565655.230481 mapwrite 0x025766 thru 0x02b772 (0x600d bytes) 000645 1786565655.312064 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2b773): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b773): Mirror 2: 0x54d429ce (stale) Mirror 3: 0x54d429ce (stale) CRC-32 checksum value for chunk [0, 0x2b773): Mirror 1: 0x7840d828 Mirror 4: 0x7840d828 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b773: skipped 000646 1786565656.487132 trunc from 0x02b773 to 0x01855e (0x13214 bytes) 000649 1786565656.661612 mapwrite 0x00e4d3 thru 0x010772 (0x22a0 bytes) 000650 1786565656.729950 mapread 0x00513c thru 0x012e8c (0xdd51 bytes) 000652 1786565656.741424 mapread 0x0107ef thru 0x01855e (0x7d70 bytes) 000654 1786565656.744565 trunc from 0x01855f to 0x0213e4 (0x8e86 bytes) 000655 1786565656.772358 write 0x03a058 thru 0x03ffff (0x5fa8 bytes) 000656 1786565656.777608 mapread 0x01bcb5 thru 0x027448 (0xb794 bytes) 000658 1786565656.809539 trunc from 0x040000 to 0x0003cb (0x3fc34 bytes) 000660 1786565656.831213 write 0x018cd5 thru 0x022868 (0x9b94 bytes) 000664 1786565656.840613 MIRROR_RESYNC 4 mirrors 000665 1786565658.688577 mapread 0x008004 thru 0x015616 (0xd613 bytes) 000666 1786565658.714222 write 0x0250fe thru 0x02dbf8 (0x8afb bytes) 000669 1786565658.771055 mapwrite 0x036972 thru 0x03dbed (0x727c bytes) 000670 1786565658.869417 MIRROR_RESYNC 4 mirrors 000671 1786565660.079667 read 0x0125c0 thru 0x01d76a (0xb1ab bytes) 000675 1786565660.096165 mapread 0x027896 thru 0x031028 (0x9793 bytes) 000676 1786565660.136694 write 0x038c60 thru 0x03eca3 (0x6044 bytes) 000677 1786565660.327178 MIRROR_RESYNC 4 mirrors 000679 1786565662.091002 mapread 0x0228c0 thru 0x02c4c3 (0x9c04 bytes) 000681 1786565662.111668 write 0x02ef2d thru 0x02f332 (0x406 bytes) 000683 1786565662.174516 write 0x02ef89 thru 0x03dc4c (0xecc4 bytes) 000685 1786565662.183138 trunc from 0x03eca4 to 0x02b594 (0x1370f bytes) 000686 1786565662.209889 trunc from 0x02b595 to 0x018038 (0x1355c bytes) 000687 1786565662.229168 write 0x039300 thru 0x0398ed (0x5ee bytes) 000688 1786565662.234521 mapread 0x00c156 thru 0x01b3c3 (0xf26e bytes) 000692 1786565662.256207 mapwrite 0x005011 thru 0x011b07 (0xcaf7 bytes) 000694 1786565662.334790 MIRROR_RESYNC 4 mirrors 000695 1786565664.436119 write 0x01d776 thru 0x02238f (0x4c1a bytes) 000697 1786565664.522908 mapread 0x00b070 thru 0x018777 (0xd708 bytes) 000698 1786565664.553202 trunc from 0x0398ee to 0x03a2c2 (0x9d5 bytes) 000699 1786565664.589106 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a2c3): Mirror 2: 0x59381f33 (stale) Mirror 3: 0x59381f33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2c3): Mirror 2: 0x59381f33 (stale) Mirror 3: 0x59381f33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2c3: skipped 000700 1786565664.987191 mapwrite 0x006926 thru 0x008cbd (0x2398 bytes) 000701 1786565665.101098 read 0x018397 thru 0x027fd0 (0xfc3a bytes) 000702 1786565665.115221 trunc from 0x03a2c3 to 0x01d251 (0x1d071 bytes) 000703 1786565665.136118 read 0x004712 thru 0x008d07 (0x45f6 bytes) 000705 1786565665.138868 mapread 0x01ce50 thru 0x01d251 (0x402 bytes) 000707 1786565665.142093 trunc from 0x01d252 to 0x02dec8 (0x10c77 bytes) 000708 1786565665.162784 write 0x033f25 thru 0x038114 (0x41f0 bytes) 000709 1786565665.165585 trunc from 0x038115 to 0x00b928 (0x2c7ec bytes) 000710 1786565665.186761 read 0x00a314 thru 0x00b928 (0x1615 bytes) 000712 1786565665.189315 mapwrite 0x00e557 thru 0x0166b7 (0x8161 bytes) 000715 1786565665.263296 read 0x01614b thru 0x0166b7 (0x56d bytes) 000716 1786565665.264199 write 0x009262 thru 0x012472 (0x9211 bytes) 000718 1786565665.268084 trunc from 0x0166b8 to 0x00a686 (0xc031 bytes) 000719 1786565665.290004 mapread 0x0041fd thru 0x00a686 (0x648a bytes) 000723 1786565665.298670 MIRROR_RESYNC 3 mirrors 000725 1786565665.967961 trunc from 0x00a687 to 0x00e9ac (0x4326 bytes) 000726 1786565666.079436 trunc from 0x00e9ad to 0x02fe16 (0x2146a bytes) 000727 1786565666.104727 mapwrite 0x00183e thru 0x010b8a (0xf34d bytes) 000729 1786565666.188382 mapwrite 0x022f0c thru 0x024e6c (0x1f61 bytes) 000730 1786565666.245392 mapread 0x02e377 thru 0x02fe16 (0x1aa0 bytes) 000735 1786565666.257053 mapread 0x02ab62 thru 0x02fe16 (0x52b5 bytes) 000738 1786565666.261173 read 0x00f9ef thru 0x016b9e (0x71b0 bytes) 000741 1786565666.283694 MIRROR_RESYNC 3 mirrors 000742 1786565667.508404 trunc from 0x02fe17 to 0x03a55b (0xa745 bytes) 000743 1786565667.614280 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a55c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a55c): Mirror 2: 0x8f43e8d4 (stale) Mirror 3: 0x8f43e8d4 (stale) CRC-32 checksum value for chunk [0, 0x3a55c): Mirror 1: 0x8f43e8d4 Mirror 4: 0x8f43e8d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a55c: skipped 000744 1786565668.771294 mapread 0x001212 thru 0x00a821 (0x9610 bytes) 000746 1786565668.881670 mapwrite 0x009e3a thru 0x00e63f (0x4806 bytes) 000748 1786565669.172055 trunc from 0x03a55c to 0x022313 (0x18248 bytes) 000749 1786565669.200245 trunc from 0x022314 to 0x02e2b6 (0xbfa3 bytes) 000751 1786565669.230018 write 0x03b3ef thru 0x03f21a (0x3e2c bytes) 000752 1786565669.233001 trunc from 0x03f21b to 0x02315a (0x1c0c0 bytes) 000753 1786565669.260039 trunc from 0x02315b to 0x02ba2e (0x88d4 bytes) 000754 1786565669.281236 trunc from 0x02ba2f to 0x019fca (0x11a64 bytes) 000755 1786565669.306469 read 0x0173df thru 0x019fca (0x2bec bytes) 000756 1786565669.313084 write 0x014109 thru 0x01bc35 (0x7b2d bytes) 000757 1786565669.325989 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1bc36): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bc36): Mirror 2: 0x69f74940 (stale) Mirror 3: 0x69f74940 (stale) Mirror 4: 0x69f74940 (stale) CRC-32 checksum value for chunk [0, 0x1bc36): Mirror 1: 0x1a3228f8 Mirror 5: 0x1a3228f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bc36: skipped 000758 1786565670.379131 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1bc36): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bc36): Mirror 2: 0x69f74940 (stale) Mirror 3: 0x69f74940 (stale) Mirror 4: 0x69f74940 (stale) CRC-32 checksum value for chunk [0, 0x1bc36): Mirror 1: 0x1a3228f8 Mirror 5: 0x1a3228f8 Mirror 6: 0x1a3228f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bc36: skipped 000759 1786565671.693886 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bc36): Mirror 2: 0x69f74940 (stale) Mirror 3: 0x69f74940 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bc36): Mirror 2: 0x69f74940 (stale) Mirror 3: 0x69f74940 (stale) CRC-32 checksum value for chunk [0, 0x1bc36): Mirror 1: 0x1a3228f8 Mirror 5: 0x1a3228f8 Mirror 6: 0x1a3228f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bc36: skipped 000760 1786565672.303604 mapread 0x01ae28 thru 0x01bc35 (0xe0e bytes) 000761 1786565672.319763 trunc from 0x01bc36 to 0x00086a (0x1b3cb bytes) 000762 1786565672.485875 write 0x035c25 thru 0x03df39 (0x8315 bytes) 000763 1786565672.487842 trunc from 0x03df3a to 0x0022be (0x3bc7b bytes) 000764 1786565672.518281 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x22bf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22bf): Mirror 2: 0x4a5d896e (stale) Mirror 3: 0x4a5d896e (stale) Mirror 5: 0x4a5d896e (stale) Mirror 6: 0x4a5d896e (stale) CRC-32 checksum value for chunk [0, 0x22bf): Mirror 1: 0x7ba64db3 Mirror 7: 0x7ba64db3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22bf: skipped 000768 1786565673.446910 read 0x00035f thru 0x0022be (0x1f60 bytes) 000769 1786565673.469997 mapread 0x000d1d thru 0x0022be (0x15a2 bytes) 000770 1786565673.489497 mapwrite 0x0034ec thru 0x007f0c (0x4a21 bytes) 000771 1786565673.695026 mapwrite 0x015177 thru 0x01a051 (0x4edb bytes) 000772 1786565673.763079 read 0x00eabf thru 0x0166c2 (0x7c04 bytes) 000773 1786565673.765918 write 0x03c676 thru 0x03ffff (0x398a bytes) 000775 1786565673.780440 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3c4981d1 (stale) Mirror 3: 0x3c4981d1 (stale) Mirror 5: 0xa7e8f807 (stale) Mirror 6: 0xa7e8f807 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3c4981d1 (stale) Mirror 3: 0x3c4981d1 (stale) Mirror 5: 0xa7e8f807 (stale) Mirror 6: 0xa7e8f807 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000779 1786565674.259658 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3c4981d1 (stale) Mirror 3: 0x3c4981d1 (stale) Mirror 5: 0xa7e8f807 (stale) Mirror 6: 0xa7e8f807 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6346bb69 Mirror 7: 0x6346bb69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000780 1786565675.319964 mapread 0x037459 thru 0x03d91e (0x64c6 bytes) 000781 1786565675.365693 write 0x004896 thru 0x008905 (0x4070 bytes) 000782 1786565675.457477 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3c4981d1 (stale) Mirror 3: 0x3c4981d1 (stale) Mirror 5: 0xa7e8f807 (stale) Mirror 6: 0xa7e8f807 (stale) Mirror 7: 0x6346bb69 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x168a5e38 Mirror 8: 0x168a5e38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000784 1786565676.792723 write 0x0274db thru 0x032ecf (0xb9f5 bytes) 000785 1786565676.887200 read 0x010940 thru 0x01c0a7 (0xb768 bytes) 000787 1786565676.901650 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3c4981d1 (stale) Mirror 3: 0x3c4981d1 (stale) Mirror 5: 0xa7e8f807 (stale) Mirror 6: 0xa7e8f807 (stale) Mirror 7: 0x6346bb69 (stale) Mirror 8: 0x168a5e38 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3c5309c1 Mirror 9: 0x3c5309c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000788 1786565678.062962 trunc from 0x040000 to 0x018cfa (0x27305 bytes) 000789 1786565678.145431 read 0x00ee0e thru 0x018cfa (0x9eed bytes) 000790 1786565678.152925 trunc from 0x018cfb to 0x011f92 (0x6d68 bytes) 000791 1786565678.167302 mapread 0x007eb6 thru 0x00c359 (0x44a4 bytes) 000793 1786565678.183756 mapwrite 0x00f0b3 thru 0x012f3b (0x3e89 bytes) 000796 1786565678.305219 mapwrite 0x008e6c thru 0x01087b (0x7a10 bytes) 000797 1786565678.364480 read 0x001f8b thru 0x00d0d8 (0xb14e bytes) 000798 1786565678.370725 mapread 0x005f6e thru 0x00d05c (0x70ef bytes) 000801 1786565678.373880 read 0x00e3b0 thru 0x012f3b (0x4b8c bytes) 000802 1786565678.378921 MIRROR_RESYNC 8 mirrors 000803 1786565679.594963 write 0x0142c7 thru 0x01e021 (0x9d5b bytes) 000805 1786565679.710511 mapread 0x008573 thru 0x015f3d (0xd9cb bytes) 000806 1786565679.750443 mapwrite 0x02d3d8 thru 0x0344d1 (0x70fa bytes) 000808 1786565679.856949 mapwrite 0x02735e thru 0x0345dd (0xd280 bytes) 000810 1786565679.937589 mapwrite 0x03fefa thru 0x03ffff (0x106 bytes) 000811 1786565680.059189 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe5827ea6 (stale) Mirror 3: 0xe5827ea6 (stale) Mirror 5: 0xe5827ea6 (stale) Mirror 6: 0xe5827ea6 (stale) Mirror 8: 0xe5827ea6 (stale) Mirror 9: 0xe5827ea6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe5827ea6 (stale) Mirror 3: 0xe5827ea6 (stale) Mirror 5: 0xe5827ea6 (stale) Mirror 6: 0xe5827ea6 (stale) Mirror 8: 0xe5827ea6 (stale) Mirror 9: 0xe5827ea6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000812 1786565680.507791 trunc from 0x040000 to 0x00ea07 (0x315f8 bytes) 000813 1786565680.535283 mapwrite 0x0126c7 thru 0x017688 (0x4fc2 bytes) 000815 1786565680.660698 mapwrite 0x039823 thru 0x03bdc2 (0x25a0 bytes) 000817 1786565680.728104 trunc from 0x03bdc3 to 0x00d216 (0x2ebac bytes) 000822 1786565680.757438 mapread 0x00a738 thru 0x00d216 (0x2adf bytes) 000828 1786565680.771501 mapread 0x008d05 thru 0x00b458 (0x2754 bytes) 000830 1786565680.774356 trunc from 0x00d217 to 0x00fff6 (0x2de0 bytes) 000831 1786565680.795472 trunc from 0x00fff7 to 0x028e4e (0x18e58 bytes) 000834 1786565680.822008 trunc from 0x028e4f to 0x01b5da (0xd874 bytes) 000835 1786565680.837963 trunc from 0x01b5db to 0x03a1d0 (0x1ebf6 bytes) 000837 1786565680.854104 mapread 0x020baa thru 0x021ecf (0x1326 bytes) 000838 1786565680.867803 write 0x03237f thru 0x03f040 (0xccc2 bytes) 000839 1786565680.874019 write 0x0326dc thru 0x035961 (0x3286 bytes) 000841 1786565680.875538 mapread 0x016c47 thru 0x017942 (0xcfc bytes) 000842 1786565680.887650 mapwrite 0x03d99e thru 0x03ffff (0x2662 bytes) 000843 1786565680.964808 trunc from 0x040000 to 0x02774b (0x188b4 bytes) 000844 1786565680.984487 write 0x02971d thru 0x031cec (0x85d0 bytes) 000847 1786565680.989364 mapread 0x024542 thru 0x02b499 (0x6f58 bytes) 000849 1786565680.996684 mapread 0x023c58 thru 0x027e78 (0x4221 bytes) 000850 1786565681.002937 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31ced): Mirror 2: 0xbc05a22f (stale) Mirror 5: 0xbc05a22f (stale) Mirror 6: 0xbc05a22f (stale) Mirror 8: 0xbc05a22f (stale) Mirror 9: 0xbc05a22f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31ced): Mirror 2: 0xbc05a22f (stale) Mirror 5: 0xbc05a22f (stale) Mirror 6: 0xbc05a22f (stale) Mirror 8: 0xbc05a22f (stale) Mirror 9: 0xbc05a22f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31ced: skipped 000852 1786565681.406625 mapread 0x012c74 thru 0x020f36 (0xe2c3 bytes) 000853 1786565681.432115 read 0x01275e thru 0x012fd6 (0x879 bytes) 000854 1786565681.433188 trunc from 0x031ced to 0x03140a (0x8e2 bytes) 000855 1786565681.451512 write 0x00e2a8 thru 0x0137bd (0x5516 bytes) 000857 1786565681.453439 write 0x024255 thru 0x024c76 (0xa22 bytes) 000858 1786565681.455468 write 0x00bd48 thru 0x01adb1 (0xf06a bytes) 000859 1786565681.461799 mapwrite 0x03d41b thru 0x03ffff (0x2be5 bytes) 000862 1786565681.538795 write 0x01ebef thru 0x02a71c (0xbb2e bytes) 000863 1786565681.545734 trunc from 0x040000 to 0x0270da (0x18f25 bytes) 000864 1786565681.564522 mapread 0x004153 thru 0x007081 (0x2f2f bytes) 000868 1786565681.576296 write 0x021625 thru 0x031572 (0xff4e bytes) 000869 1786565681.578544 write 0x006ccb thru 0x00efe3 (0x8319 bytes) 000874 1786565681.586554 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31573): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31573): Mirror 2: 0xfda45436 (stale) Mirror 5: 0xfda45436 (stale) Mirror 6: 0xfda45436 (stale) Mirror 8: 0xfda45436 (stale) Mirror 9: 0xfda45436 (stale) CRC-32 checksum value for chunk [0, 0x31573): Mirror 1: 0x48a6dca8 Mirror 10: 0x48a6dca8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31573: skipped 000875 1786565682.470002 mapwrite 0x001126 thru 0x00d768 (0xc643 bytes) 000876 1786565682.705377 read 0x00bf25 thru 0x00fd7b (0x3e57 bytes) 000878 1786565682.718311 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x31573): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31573): Mirror 2: 0xfda45436 (stale) Mirror 5: 0xfda45436 (stale) Mirror 6: 0xfda45436 (stale) Mirror 8: 0xfda45436 (stale) Mirror 9: 0xfda45436 (stale) Mirror 10: 0x48a6dca8 (stale) CRC-32 checksum value for chunk [0, 0x31573): Mirror 1: 0xa3f86e46 Mirror 11: 0xa3f86e46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31573: skipped 000879 1786565683.594161 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x31573): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31573): Mirror 2: 0xfda45436 (stale) Mirror 5: 0xfda45436 (stale) Mirror 6: 0xfda45436 (stale) Mirror 8: 0xfda45436 (stale) Mirror 9: 0xfda45436 (stale) Mirror 10: 0x48a6dca8 (stale) CRC-32 checksum value for chunk [0, 0x31573): Mirror 1: 0xa3f86e46 Mirror 11: 0xa3f86e46 Mirror 12: 0xa3f86e46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31573: skipped 000882 1786565684.631598 write 0x033231 thru 0x03ffff (0xcdcf bytes) 000884 1786565684.811322 write 0x03b5b3 thru 0x03e698 (0x30e6 bytes) 000885 1786565684.813697 read 0x0276af thru 0x0326a1 (0xaff3 bytes) 000886 1786565684.833356 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe5827ea6 (stale) Mirror 6: 0xe5827ea6 (stale) Mirror 8: 0xe5827ea6 (stale) Mirror 9: 0xe5827ea6 (stale) Mirror 10: 0x418922ec (stale) Mirror 11: 0xb9304a8e (stale) Mirror 12: 0xb9304a8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe5827ea6 (stale) Mirror 6: 0xe5827ea6 (stale) Mirror 8: 0xe5827ea6 (stale) Mirror 9: 0xe5827ea6 (stale) Mirror 10: 0x418922ec (stale) Mirror 11: 0xb9304a8e (stale) Mirror 12: 0xb9304a8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000887 1786565685.573816 trunc from 0x040000 to 0x003cd0 (0x3c32f bytes) 000890 1786565685.592008 write 0x032a96 thru 0x03a349 (0x78b4 bytes) 000891 1786565685.597431 read 0x00cbf2 thru 0x00d5fd (0xa0c bytes) 000892 1786565685.610262 MIRROR_RESYNC 8 mirrors 000897 1786565688.398388 trunc from 0x03a34a to 0x032849 (0x7b00 bytes) 000898 1786565688.553261 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3284a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3284a): Mirror 5: 0xbd294f32 (stale) Mirror 6: 0xbd294f32 (stale) Mirror 8: 0xbd294f32 (stale) Mirror 9: 0xbd294f32 (stale) Mirror 10: 0xbd294f32 (stale) Mirror 11: 0xbd294f32 (stale) Mirror 12: 0xbd294f32 (stale) CRC-32 checksum value for chunk [0, 0x3284a): Mirror 1: 0xbd294f32 Mirror 13: 0xbd294f32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3284a: skipped 000899 1786565689.728631 trunc from 0x03284a to 0x00e118 (0x24731 bytes) 000900 1786565689.858982 trunc from 0x00e119 to 0x016912 (0x87fa bytes) 000901 1786565689.898922 mapwrite 0x036dea thru 0x03ffff (0x9216 bytes) 000902 1786565690.008629 read 0x01ff15 thru 0x023503 (0x35ef bytes) 000903 1786565690.019392 mapread 0x03efc8 thru 0x03fd64 (0xd9d bytes) 000904 1786565690.021530 mapread 0x02b4a5 thru 0x02f730 (0x428c bytes) 000909 1786565690.042499 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x578e5952 (stale) Mirror 6: 0x578e5952 (stale) Mirror 8: 0x578e5952 (stale) Mirror 9: 0x578e5952 (stale) Mirror 11: 0x578e5952 (stale) Mirror 12: 0x578e5952 (stale) Mirror 13: 0x99023041 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x578e5952 (stale) Mirror 6: 0x578e5952 (stale) Mirror 8: 0x578e5952 (stale) Mirror 9: 0x578e5952 (stale) Mirror 11: 0x578e5952 (stale) Mirror 12: 0x578e5952 (stale) Mirror 13: 0x99023041 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000910 1786565690.630554 trunc from 0x040000 to 0x03be4f (0x41b0 bytes) 000911 1786565690.666603 write 0x022412 thru 0x02d386 (0xaf75 bytes) 000913 1786565690.686026 trunc from 0x03be50 to 0x03f3a7 (0x3558 bytes) 000914 1786565690.705603 write 0x0312df thru 0x03c3c0 (0xb0e2 bytes) 000915 1786565690.718467 mapwrite 0x009c19 thru 0x013af8 (0x9ee0 bytes) 000916 1786565690.805741 mapread 0x02571e thru 0x031b68 (0xc44b bytes) 000919 1786565690.815939 mapread 0x038b92 thru 0x03f3a7 (0x6816 bytes) 000920 1786565690.825076 trunc from 0x03f3a8 to 0x00b544 (0x33e63 bytes) 000921 1786565690.859129 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb545): Mirror 5: 0x63d65c10 (stale) Mirror 8: 0x63d65c10 (stale) Mirror 9: 0x63d65c10 (stale) Mirror 11: 0x63d65c10 (stale) Mirror 12: 0x63d65c10 (stale) Mirror 13: 0x63d65c10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb545): Mirror 5: 0x63d65c10 (stale) Mirror 8: 0x63d65c10 (stale) Mirror 9: 0x63d65c10 (stale) Mirror 11: 0x63d65c10 (stale) Mirror 12: 0x63d65c10 (stale) Mirror 13: 0x63d65c10 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb545: skipped 000922 1786565691.233175 mapread 0x0038a4 thru 0x0045e6 (0xd43 bytes) 000924 1786565691.253765 write 0x012ba5 thru 0x01ea89 (0xbee5 bytes) 000927 1786565691.259126 mapwrite 0x018628 thru 0x01a711 (0x20ea bytes) 000929 1786565691.360392 write 0x032fc1 thru 0x03b68a (0x86ca bytes) 000930 1786565691.369511 mapread 0x02a415 thru 0x02cf09 (0x2af5 bytes) 000932 1786565691.409648 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b68b): Mirror 5: 0x1c2b89b9 (stale) Mirror 8: 0x1c2b89b9 (stale) Mirror 9: 0x1c2b89b9 (stale) Mirror 11: 0x1c2b89b9 (stale) Mirror 13: 0x6f52773d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b68b): Mirror 5: 0x1c2b89b9 (stale) Mirror 8: 0x1c2b89b9 (stale) Mirror 9: 0x1c2b89b9 (stale) Mirror 11: 0x1c2b89b9 (stale) Mirror 13: 0x6f52773d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b68b: skipped 000933 1786565692.063449 MIRROR_RESYNC 6 mirrors 000935 1786565695.789664 write 0x02571c thru 0x031eb4 (0xc799 bytes) 000937 1786565695.925601 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b68b): Mirror 5: 0xa4d3556f (stale) Mirror 8: 0xa4d3556f (stale) Mirror 11: 0xa4d3556f (stale) Mirror 13: 0xa4d3556f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b68b): Mirror 5: 0xa4d3556f (stale) Mirror 8: 0xa4d3556f (stale) Mirror 11: 0xa4d3556f (stale) Mirror 13: 0xa4d3556f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b68b: skipped 000938 1786565696.421899 read 0x01076e thru 0x01ee62 (0xe6f5 bytes) 000939 1786565696.431537 read 0x01c804 thru 0x024308 (0x7b05 bytes) 000940 1786565696.455394 read 0x01f187 thru 0x02cf4a (0xddc4 bytes) 000943 1786565696.466652 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b68b): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b68b): Mirror 5: 0xa4d3556f (stale) Mirror 8: 0xa4d3556f (stale) Mirror 11: 0xa4d3556f (stale) Mirror 13: 0xa4d3556f (stale) CRC-32 checksum value for chunk [0, 0x3b68b): Mirror 1: 0xca346988 Mirror 14: 0xca346988 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b68b: skipped 000944 1786565698.325571 trunc from 0x03b68b to 0x026cb2 (0x149d8 bytes) 000945 1786565698.438490 mapwrite 0x03704b thru 0x03ffff (0x8fb5 bytes) 000947 1786565698.577287 read 0x026d25 thru 0x02965d (0x2939 bytes) 000948 1786565698.586189 write 0x004e18 thru 0x011231 (0xc41a bytes) 000950 1786565698.610557 write 0x0119f3 thru 0x02180a (0xfe18 bytes) 000954 1786565698.628152 trunc from 0x040000 to 0x02a6fc (0x15903 bytes) 000955 1786565698.652124 write 0x023fcd thru 0x030186 (0xc1ba bytes) 000959 1786565698.662964 trunc from 0x030187 to 0x01b9bf (0x147c7 bytes) 000962 1786565698.690431 mapwrite 0x009418 thru 0x011ed1 (0x8aba bytes) 000964 1786565698.751712 MIRROR_RESYNC 6 mirrors 000966 1786565699.890495 mapwrite 0x02637a thru 0x02c063 (0x5cea bytes) 000967 1786565700.152538 trunc from 0x02c064 to 0x0360b1 (0xa04e bytes) 000968 1786565700.184076 trunc from 0x0360b2 to 0x027deb (0xe2c6 bytes) 000969 1786565700.221363 mapread 0x00c227 thru 0x00f14d (0x2f27 bytes) 000970 1786565700.238354 trunc from 0x027dec to 0x02e4f5 (0x670a bytes) 000971 1786565700.272782 trunc from 0x02e4f6 to 0x03cdac (0xe8b7 bytes) 000973 1786565700.290952 trunc from 0x03cdad to 0x03a016 (0x2d96 bytes) 000974 1786565700.308464 trunc from 0x03a017 to 0x021742 (0x188d4 bytes) 000975 1786565700.326703 MIRROR_RESYNC 6 mirrors 000976 1786565701.610808 write 0x036601 thru 0x03ffff (0x99ff bytes) 000977 1786565701.679385 read 0x03996e thru 0x03ffff (0x6692 bytes) 000979 1786565701.680680 read 0x02e98d thru 0x032420 (0x3a94 bytes) 000984 1786565701.695513 trunc from 0x040000 to 0x038da0 (0x725f bytes) 000992 1786565701.726057 mapread 0x037014 thru 0x038da0 (0x1d8d bytes) 000993 1786565701.729079 mapread 0x0043ae thru 0x0086bd (0x4310 bytes) 000995 1786565701.732008 read 0x02e402 thru 0x03613e (0x7d3d bytes) 000996 1786565701.733124 trunc from 0x038da1 to 0x02f5f8 (0x97a8 bytes) 000999 1786565701.752382 read 0x01c04c thru 0x029917 (0xd8cc bytes) 001000 1786565701.753286 write 0x006226 thru 0x00b886 (0x5661 bytes) All operations completed A-OK! PASS 70a (119s) == sanity-flr test 71: check mirror extend parent fid ==== 16:15:08 (1786565708) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 16:15:11 (1786565711) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.426071 s, 4.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18125 s, 5.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 16:15:23 (1786565723) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre2 Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3342016 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7392768 .. Extending file size to 4605792 .. failed Extending file size to 6328384 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8863264 .. Extending file size to 6331520 .. Extending file size to 8752224 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8993504 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8994944 .. Waiting 57538 57539 57542 57544 57545 done 192.168.201.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre2 (opts:) 192.168.201.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:16:45 (1786565805) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre2 Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8908384 .. done Extending file size to 8287936 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9041408 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9061440 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8908704 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9298880 .. Waiting 59008 59010 59011 59013 59014 done 192.168.201.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre2 (opts:) 192.168.201.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:18:06 (1786565886) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104021 s, 20.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.511397 s, 4.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf836b8ba (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:18:27 (1786565907) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0865716 s, 24.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 16:18:34 (1786565914) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f2:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 0x30001 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:18:42 (1786565922) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.249622 s, 8.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:18:51 (1786565931) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.232001 s, 9.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:19:00 (1786565940) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.410533 s, 5.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:19:32 (1786565972) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg157-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137494 s, 7.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:20:28 (1786566028) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg157-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.310139 s, 6.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:21:28 (1786566088) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg157-server mds-ost sync done. Creating new pool test_204f oleg157-server: Pool lustre.test_204f created Adding targets to pool oleg157-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg157-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.368921 s, 5.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg157-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg157-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg157-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:22:47 (1786566167) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:22:56 (1786566176) lfs mirror mirror: Invalid layout: Wrong flags error: lfs extend: /mnt/lustre/f205b.sanity-flr: cannot create volatile file: Invalid argument sanity-flr test_205b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs mirror extend -N --flags=nocompr /mnt/lustre/f205b.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4329:test_205b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4346:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_205b.*.1786566180.log rsync: chown "/tmp/testlogs/.sanity-flr.test_205b.debug_log.oleg157-server.1786566180.log.EyC7Wi" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_205b.dmesg.oleg157-server.1786566180.log.Hc2HEA" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:23:14 (1786566194) oleg157-server: Pool lustre.test_206 created Adding targets to pool oleg157-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg157-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg157-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg157-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg157-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:23:41 (1786566221) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991921 s, 10.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717679 s, 14.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8488eaee Mirror 3: 0x8488eaee PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:23:52 (1786566232) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:23:56 (1786566236) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:24:00 (1786566240) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.161597 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1150000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb30000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb30000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb30000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xf00000 Read this (4 bytes): BBBB 192.168.201.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:24:36 (1786566276) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0279607 s, 293 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:24:43 (1786566283) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291404 s, 36.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834453 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-12 16:24:44.000000000 -0400 Modify: 2026-08-12 16:24:44.000000000 -0400 Change: 2026-08-12 16:24:45.000000000 -0400 Birth: 2026-08-12 16:24:44.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:24:50 (1786566290) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:24:53 (1786566293) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.602347 s, 68.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0447236 s, 91.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 12 16:24 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x217:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 12 16:24 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:25:01 (1786566301) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:25:08 (1786566308) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1830 sec =========== 16:25:17 (1786566317) sanity-flr: FAIL: test_0m failed to call extend /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr sanity-flr: FAIL: test_205b /home/green/git/lustre-release/lustre/utils/lfs mirror extend -N --flags=nocompr /mnt/lustre/f205b.sanity-flr failed === sanity-flr: start cleanup 16:25:18 (1786566318) === === sanity-flr: finish cleanup 16:25:24 (1786566324) ===