-----============= acceptance-small: sanity-flr ============----- Thu Sep 17 04:55:09 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg215-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 04:55:31 (1789635331) === oleg215-client.virtnet: executing check_config_client /mnt/lustre oleg215-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg215-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff931905f6a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff931905f6a000.idle_timeout=debug disable quota as required oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 04:56:17 (1789635377) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8244] == sanity-flr test 0a: lfs mirror create with -N option == 04:56:20 (1789635380) PASS 0a (29s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:56:49 (1789635409) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:56:53 (1789635413) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (7s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:57:00 (1789635420) PASS 0d (34s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:57:35 (1789635455) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:57:41 (1789635461) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 04:57:46 (1789635466) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff931905f6a000.stats=clear osc.lustre-OST0001-osc-ffff931905f6a000.stats=clear PASS 0g (13s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:57:59 (1789635479) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:58:11 (1789635491) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:58:22 (1789635502) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:58:27 (1789635507) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (12s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:58:39 (1789635519) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 04:58:51 (1789635531) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:59:00 (1789635540) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:59:09 (1789635549) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:59:17 (1789635557) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:59:28 (1789635568) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:59:36 (1789635576) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.458337 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891148 s, 11.8 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:59:46 (1789635586) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0907051 s, 23.1 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:59:54 (1789635594) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:00:06 (1789635606) Stopping /mnt/lustre-ost1 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff931905f6a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931905f6a000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg215-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff931905f6a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff931905f6a000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg215-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (84s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:01:30 (1789635690) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.63108 s, 13.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff931905f6a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931905f6a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg215-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:02:15 (1789635735) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff931905f6a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931905f6a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg215-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff931905f6a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff931905f6a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg215-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:03:29 (1789635809) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.26874 s, 23.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 17 05:03 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.238s user 0m0.000s sys 0m0.181s Stopping /mnt/lustre-ost1 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff931905f6a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931905f6a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.796s user 0m0.002s sys 0m0.179s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg215-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff931905f6a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff931905f6a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.824s user 0m0.001s sys 0m0.170s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg215-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg215-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff931905f6a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff931905f6a000.ost_server_uuid in FULL state after 0 sec PASS 33b (77s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:04:46 (1789635886) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:04:49 (1789635889) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:04:52 (1789635892) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 05:04:54 (1789635894) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 05:05:01 (1789635901) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.371865 s, 33.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:05:12 (1789635912) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521054 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743945 s, 14.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:05:19 (1789635919) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:05:27 (1789635927) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00436818 s, 234 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00110013 s, 931 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000746254 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00117392 s, 872 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 05:05:36 (1789635936) create test files with size 15760136 .. d89b4225743f8370462a53ab4212bde3 - 06491d93c694f70ae0e81beb97201e4b - e8a84b0c3de74d625e760e2a16781efc - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg215-client.virtnet /mnt/lustre (opts:) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 05:05:57 (1789635957) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:06:10 (1789635970) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:06:16 (1789635976) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688499 s, 15.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332342 s, 31.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 05:06:23 (1789635983) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104114 s, 20.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.292734 s, 14.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149583 s, 14.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 05:06:36 (1789635996) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 05:06:39 (1789635999) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:06:42 (1789636002) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294781 s, 35.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 05:06:49 (1789636009) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.365951 s, 8.6 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:06:57 (1789636017) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:07:06 (1789636026) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.291958 s, 35.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg215-client.virtnet /mnt/lustre (opts:) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:07:16 (1789636036) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9319057b4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.36775 s, 28.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg215-client.virtnet /mnt/lustre (opts:) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff931906ee2000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:07:31 (1789636051) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:08:06 (1789636086) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (29s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:08:35 (1789636115) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 05:08:38 (1789636118) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.189387 s, 16.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.147387 s, 21.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x499cdd45 (nosync,stale) Mirror 2: 0x499cdd45 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5462a0c088674021ec42e0f1186d456d - mirror 1 checksum: f21f54c863f257f47f727baa66210fe7 - mirror 2 checksum: 5462a0c088674021ec42e0f1186d456d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-17T05:08:40-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-17T05:08:40-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5462a0c088674021ec42e0f1186d456d - mirror 1 checksum: f21f54c863f257f47f727baa66210fe7 - mirror 2 checksum: f21f54c863f257f47f727baa66210fe7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-17T05:08:40-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-17T05:08:40-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 05:08:47 (1789636127) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281984 s, 37.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:08:53 (1789636133) Failing mds1 on oleg215-server Stopping /mnt/lustre-mds1 (opts:) on oleg215-server 05:09:04 (1789636144) shut down facet: mds1 facet_host: oleg215-server facet_failover_host: oleg215-server Failover mds1 to oleg215-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 05:09:25 (1789636165) targets are mounted 05:09:25 (1789636165) facet_failover done oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310295 s, 33.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 50A (46s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:09:39 (1789636179) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.030632 s, 669 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0188992 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:09:47 (1789636187) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0328432 s, 624 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00855514 s, 2.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:09:55 (1789636195) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0036434 s, 4.5 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:10:02 (1789636202) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0683878 s, 299 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:10:08 (1789636208) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00594083 s, 689 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0363878 s, 113 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:10:14 (1789636214) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) PASS 60b (16s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:10:30 (1789636230) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:10:56 (1789636256) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:11:18 (1789636278) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:11:40 (1789636300) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 05:11:45 (1789636305) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57588 mirror split pid 57590 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57588 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57590 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 05:12:50 (1789636370) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6372 skipping zero length zero range skipping zero size read 000003 1789636373.200809 mapwrite 0x00b10d thru 0x0139a6 (0x889a bytes) 000005 1789636373.289414 mapwrite 0x03cdcf thru 0x03ffff (0x3231 bytes) truncating to largest ever: 0xe978 000006 1789636373.363300 trunc from 0x040000 to 0x00e977 (0x31688 bytes) 000007 1789636373.384501 write 0x017634 thru 0x01f2a2 (0x7c6f bytes) zero_range to largest ever: 0x1e333 000008 1789636373.386175 trunc from 0x01cf6a to 0x01e332 (0x13c9 bytes) 000009 1789636373.407235 fallocate 0x00d814 thru 0x012b81 (0x536e bytes) 000011 1789636373.414269 punch from 0x01724f to 0x01dbc3 (0x6975 bytes) 000012 1789636373.421991 mapread 0x014200 thru 0x017088 (0x2e89 bytes) 000013 1789636373.436518 write 0x004ef3 thru 0x00693a (0x1a48 bytes) truncating to largest ever: 0x3f893 000014 1789636373.442344 trunc from 0x01f2a3 to 0x03f892 (0x205f0 bytes) 000015 1789636373.465269 read 0x017492 thru 0x01fd5b (0x88ca bytes) 000016 1789636373.473974 trunc from 0x03f893 to 0x039e85 (0x5a0d bytes) 000017 1789636373.490089 write 0x034148 thru 0x03a577 (0x6430 bytes) 000018 1789636373.497108 read 0x004ea8 thru 0x006113 (0x126c bytes) 000020 1789636373.498934 fallocate 0x0047ce thru 0x009804 (0x5037 bytes) 000022 1789636373.506355 trunc from 0x009410 to 0x00da34 (0x4625 bytes) 000025 1789636373.512493 punch from 0x0218bb to 0x031246 (0xf98c bytes) 000026 1789636373.517247 trunc from 0x0080a9 to 0x00d4a2 (0x53fa bytes) 000027 1789636373.521379 read 0x01d9f7 thru 0x026d35 (0x933f bytes) 000028 1789636373.536581 MIRROR_EXTEND to 2 mirrors 000030 1789636374.515590 mapwrite 0x02473c thru 0x02a68a (0x5f4f bytes) 000031 1789636374.674964 trunc from 0x01b726 to 0x01cbad (0x1488 bytes) 000032 1789636374.688714 read 0x02296f thru 0x0289b4 (0x6046 bytes) 000033 1789636374.692566 mapwrite 0x038887 thru 0x03ffff (0x7779 bytes) 000034 1789636374.779344 mapread 0x013610 thru 0x01ea97 (0xb488 bytes) 000036 1789636374.806887 trunc from 0x000389 to 0x004b1f (0x4797 bytes) 000037 1789636374.813287 fallocate 0x036456 thru 0x036573 (0x11e bytes) 000038 1789636374.818762 trunc from 0x040000 to 0x02a01d (0x15fe2 bytes) 000039 1789636374.840269 mapwrite 0x0101c4 thru 0x01886d (0x86aa bytes) 000040 1789636374.860217 fallocate 0x01384c thru 0x01f658 (0xbe0d bytes) 000041 1789636374.874545 MIRROR_RESYNC 2 mirrors 000042 1789636375.659017 mapread 0x017524 thru 0x0227d5 (0xb2b2 bytes) 000043 1789636375.685490 trunc from 0x010e6a to 0x012a2d (0x1bc4 bytes) 000044 1789636375.719098 trunc from 0x02a01e to 0x024ff1 (0x502c bytes) 000045 1789636375.750879 trunc from 0x024ff2 to 0x022b7a (0x2477 bytes) 000046 1789636375.777897 trunc from 0x022b7b to 0x0068b1 (0x1c2c9 bytes) 000047 1789636375.806600 mapwrite 0x0355c8 thru 0x03ffff (0xaa38 bytes) 000049 1789636375.869751 mapread 0x0171de thru 0x026f6b (0xfd8e bytes) 000050 1789636375.884276 trunc from 0x040000 to 0x0199e9 (0x26616 bytes) 000051 1789636375.901347 mapread 0x016aa4 thru 0x0192d3 (0x2830 bytes) 000052 1789636375.902600 punch from 0x0120c6 to 0x0199e9 (0x7924 bytes) 000054 1789636375.908363 mapwrite 0x00341a thru 0x008298 (0x4e7f bytes) 000055 1789636375.938484 write 0x016743 thru 0x0212c3 (0xab81 bytes) 000056 1789636375.945310 trunc from 0x0212c4 to 0x01f80c (0x1ab7 bytes) 000057 1789636375.961950 fallocate 0x0276b1 thru 0x028fb6 (0x1906 bytes) 000058 1789636375.972979 trunc from 0x024b58 to 0x025bd6 (0x107f bytes) 000059 1789636375.980633 fallocate 0x028398 thru 0x036e65 (0xeace bytes) 000060 1789636375.985764 mapread 0x00e0e7 thru 0x01b0f8 (0xd012 bytes) 000062 1789636375.992229 punch from 0x00058d to 0x00d540 (0xcfb4 bytes) 000063 1789636375.997842 trunc from 0x01e7fb to 0x02854a (0x9d50 bytes) 000064 1789636376.016495 mapread 0x035614 thru 0x03653b (0xf28 bytes) 000065 1789636376.027348 read 0x007702 thru 0x00aa03 (0x3302 bytes) 000066 1789636376.032441 punch from 0x030d5e to 0x032542 (0x17e5 bytes) 000067 1789636376.037691 punch from 0x0328c1 to 0x033953 (0x1093 bytes) 000068 1789636376.041984 trunc from 0x03007f to 0x03178e (0x1710 bytes) 000069 1789636376.050649 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000070 1789636376.249767 fallocate 0x000c53 thru 0x0093c0 (0x876e bytes) 000071 1789636376.256775 write 0x00e978 thru 0x014e8d (0x6516 bytes) 000073 1789636376.275600 write 0x03136f thru 0x03f5c5 (0xe257 bytes) 000074 1789636376.281959 punch from 0x01591a to 0x01d2e2 (0x79c9 bytes) 000075 1789636376.287550 mapread 0x013c4b thru 0x021ffa (0xe3b0 bytes) 000076 1789636376.317117 mapread 0x024605 thru 0x02a0df (0x5adb bytes) 000077 1789636376.327288 fallocate 0x02ea13 thru 0x03b6fb (0xcce9 bytes) 000078 1789636376.331367 punch from 0x0010dc to 0x004c5e (0x3b83 bytes) 000079 1789636376.335646 mapwrite 0x01dc92 thru 0x025aea (0x7e59 bytes) 000080 1789636376.355277 punch from 0x01ab21 to 0x024392 (0x9872 bytes) 000081 1789636376.361796 fallocate 0x036647 thru 0x03c1cb (0x5b85 bytes) 000082 1789636376.366998 mapread 0x031b78 thru 0x03f5c5 (0xda4e bytes) 000083 1789636376.367846 fallocate 0x0044df thru 0x009ec4 (0x59e6 bytes) 000084 1789636376.372023 trunc from 0x018c15 to 0x01ca2c (0x3e18 bytes) 000085 1789636376.377000 mapread 0x008de7 thru 0x012fc7 (0xa1e1 bytes) 000086 1789636376.385342 punch from 0x0239bd to 0x0256f7 (0x1d3b bytes) 000089 1789636376.391896 mapwrite 0x00677d thru 0x00b738 (0x4fbc bytes) 000090 1789636376.409673 trunc from 0x03f5c6 to 0x0276f3 (0x17ed2 bytes) 000091 1789636376.424340 trunc from 0x0276f4 to 0x01e013 (0x96e0 bytes) 000092 1789636376.437972 punch from 0x013e22 to 0x01bf4e (0x812d bytes) fallocating to largest ever: 0x40000 000094 1789636376.444523 fallocate 0x036f9e thru 0x03ffff (0x9062 bytes) 000095 1789636376.450018 mapwrite 0x03dac4 thru 0x03ffff (0x253c bytes) 000097 1789636376.477780 fallocate 0x00aa2a thru 0x00aed5 (0x4ac bytes) 000098 1789636376.482390 trunc from 0x03b0df to 0x03ffff (0x4f21 bytes) 000099 1789636376.487158 mapread 0x032a7b thru 0x03b244 (0x87ca bytes) 000100 1789636376.503104 fallocate 0x0393e6 thru 0x03e6c1 (0x52dc bytes) 000101 1789636376.508018 mapread 0x0065ae thru 0x0066e8 (0x13b bytes) 000102 1789636376.509313 write 0x0113b8 thru 0x01420f (0x2e58 bytes) 000103 1789636376.516986 write 0x015b53 thru 0x01c50a (0x69b8 bytes) 000104 1789636376.518598 fallocate 0x00f893 thru 0x018220 (0x898e bytes) 000106 1789636376.524250 mapread 0x00283d thru 0x0111cc (0xe990 bytes) 000107 1789636376.525350 punch from 0x01489a to 0x022581 (0xdce8 bytes) 000108 1789636376.542582 mapread 0x02c337 thru 0x032296 (0x5f60 bytes) 000109 1789636376.543850 write 0x039786 thru 0x039c92 (0x50d bytes) 000110 1789636376.545696 fallocate 0x01a260 thru 0x01c784 (0x2525 bytes) 000111 1789636376.552607 mapread 0x020f66 thru 0x02b672 (0xa70d bytes) 000112 1789636376.567908 trunc from 0x0327d4 to 0x037f3a (0x5767 bytes) 000114 1789636376.580286 write 0x039ee1 thru 0x03d93f (0x3a5f bytes) 000115 1789636376.581908 mapwrite 0x024f03 thru 0x0250ce (0x1cc bytes) 000116 1789636376.621955 mapread 0x01e32e thru 0x0205b5 (0x2288 bytes) 000117 1789636376.623086 fallocate 0x03ddd4 thru 0x03ffff (0x222c bytes) 000118 1789636376.635021 read 0x03572a thru 0x036d70 (0x1647 bytes) 000119 1789636376.654391 MIRROR_EXTEND to 2 mirrors 000120 1789636377.393834 mapwrite 0x030c59 thru 0x0361d8 (0x5580 bytes) 000121 1789636377.491883 trunc from 0x026b36 to 0x02ab91 (0x405c bytes) 000122 1789636377.510273 trunc from 0x00c022 to 0x00c76b (0x74a bytes) 000123 1789636377.516583 trunc from 0x040000 to 0x03fa06 (0x5f9 bytes) 000124 1789636377.543632 write 0x001225 thru 0x0040dc (0x2eb8 bytes) 000125 1789636377.556012 punch from 0x01137a to 0x019b96 (0x881d bytes) 000126 1789636377.563122 write 0x001b61 thru 0x0041a1 (0x2641 bytes) 000127 1789636377.564020 punch from 0x00f025 to 0x01c554 (0xd530 bytes) 000128 1789636377.569998 mapwrite 0x03d32e thru 0x03ffff (0x2cd2 bytes) 000129 1789636377.616162 punch from 0x009a72 to 0x00deaf (0x443e bytes) 000130 1789636377.625496 punch from 0x03d3a4 to 0x03ffff (0x2c5c bytes) 000131 1789636377.653647 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000132 1789636377.806016 trunc from 0x028539 to 0x02aa55 (0x251d bytes) 000133 1789636377.811745 trunc from 0x02be83 to 0x037cb0 (0xbe2e bytes) 000134 1789636377.817380 read 0x0253aa thru 0x02c61a (0x7271 bytes) 000135 1789636377.830946 trunc from 0x024ee3 to 0x02f878 (0xa996 bytes) 000136 1789636377.838847 mapread 0x00e882 thru 0x013596 (0x4d15 bytes) 000137 1789636377.858808 mapwrite 0x01a9fc thru 0x0268ff (0xbf04 bytes) 000139 1789636377.901683 read 0x005d4a thru 0x00ec9b (0x8f52 bytes) 000140 1789636377.910653 punch from 0x03fb69 to 0x03ffff (0x497 bytes) 000141 1789636377.914711 trunc from 0x040000 to 0x02a46c (0x15b93 bytes) 000144 1789636377.938845 read 0x018ff9 thru 0x01e559 (0x5561 bytes) 000145 1789636377.939648 trunc from 0x02a46d to 0x016618 (0x13e54 bytes) 000146 1789636377.957600 trunc from 0x016619 to 0x01e829 (0x8211 bytes) 000147 1789636377.975195 mapwrite 0x00e1ff thru 0x0143ce (0x61d0 bytes) 000148 1789636377.994803 mapwrite 0x032f77 thru 0x03b1ff (0x8289 bytes) 000149 1789636378.072259 mapwrite 0x02feed thru 0x032f22 (0x3036 bytes) 000150 1789636378.084371 mapread 0x01f74e thru 0x028f6e (0x9821 bytes) 000151 1789636378.089894 MIRROR_EXTEND to 2 mirrors 000152 1789636378.657470 mapwrite 0x03c4fd thru 0x03ffff (0x3b03 bytes) 000153 1789636378.768692 trunc from 0x040000 to 0x03e6aa (0x1955 bytes) 000154 1789636378.789865 trunc from 0x0126c4 to 0x012cbe (0x5fb bytes) 000156 1789636378.800756 read 0x03a646 thru 0x03e6aa (0x4065 bytes) 000157 1789636378.801536 punch from 0x039ff8 to 0x03e6aa (0x46b3 bytes) 000158 1789636378.807787 mapwrite 0x010dd5 thru 0x01cc3b (0xbe67 bytes) 000159 1789636378.855342 write 0x017c12 thru 0x018861 (0xc50 bytes) 000162 1789636378.863425 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000163 1789636379.007073 trunc from 0x03e6ab to 0x01c22b (0x2247f bytes) 000165 1789636379.028899 trunc from 0x01c22c to 0x03f82d (0x23602 bytes) 000166 1789636379.045802 mapwrite 0x0271c2 thru 0x030159 (0x8f98 bytes) 000167 1789636379.084758 mapread 0x001aaa thru 0x00a397 (0x88ee bytes) 000168 1789636379.096136 punch from 0x030159 to 0x035e2e (0x5cd6 bytes) 000169 1789636379.100881 trunc from 0x03f82e to 0x025a13 (0x19e1a bytes) 000171 1789636379.114410 trunc from 0x025a14 to 0x021c1e (0x3df5 bytes) 000172 1789636379.130422 write 0x00385a thru 0x004f49 (0x16f0 bytes) 000173 1789636379.131426 write 0x01ffeb thru 0x02792f (0x7945 bytes) 000174 1789636379.137574 mapwrite 0x015092 thru 0x020dda (0xbd49 bytes) 000175 1789636379.177221 punch from 0x0093e2 to 0x00bfaf (0x2bce bytes) 000176 1789636379.182406 trunc from 0x00fa04 to 0x01ec12 (0xf20f bytes) 000177 1789636379.187472 fallocate 0x007cfc thru 0x012e85 (0xb18a bytes) 000179 1789636379.191977 fallocate 0x0078cc thru 0x00b533 (0x3c68 bytes) 000180 1789636379.197788 mapread 0x01734a thru 0x0241e4 (0xce9b bytes) 000181 1789636379.214289 trunc from 0x01f3e1 to 0x0215d5 (0x21f5 bytes) 000182 1789636379.219632 fallocate 0x03d21e thru 0x03ffff (0x2de2 bytes) 000184 1789636379.227703 trunc from 0x002406 to 0x009d5f (0x795a bytes) 000185 1789636379.233226 read 0x00924b thru 0x00e078 (0x4e2e bytes) 000186 1789636379.241370 trunc from 0x040000 to 0x009218 (0x36de7 bytes) 000187 1789636379.256791 trunc from 0x009219 to 0x038899 (0x2f681 bytes) 000188 1789636379.273191 punch from 0x01667d to 0x021c29 (0xb5ad bytes) 000189 1789636379.277890 trunc from 0x03889a to 0x03e0e3 (0x584a bytes) 000190 1789636379.304395 read 0x015c42 thru 0x01ed17 (0x90d6 bytes) 000191 1789636379.317431 write 0x034d02 thru 0x03ffff (0xb2fe bytes) 000192 1789636379.319318 trunc from 0x00e8b6 to 0x01a94b (0xc096 bytes) 000193 1789636379.327653 mapwrite 0x01dea2 thru 0x02c346 (0xe4a5 bytes) 000194 1789636379.361739 trunc from 0x01493a to 0x022097 (0xd75e bytes) 000195 1789636379.370558 trunc from 0x02ec46 to 0x031727 (0x2ae2 bytes) 000196 1789636379.393999 mapread 0x02dc6f thru 0x0329a6 (0x4d38 bytes) 000197 1789636379.423619 punch from 0x01810c to 0x019038 (0xf2d bytes) 000199 1789636379.430993 punch from 0x01185e to 0x0178c4 (0x6067 bytes) 000200 1789636379.435722 write 0x0234a3 thru 0x02a790 (0x72ee bytes) 000202 1789636379.437711 read 0x0082a5 thru 0x00e223 (0x5f7f bytes) 000203 1789636379.448433 write 0x0186ef thru 0x0276b2 (0xefc4 bytes) 000204 1789636379.449843 read 0x023cbd thru 0x02c73e (0x8a82 bytes) 000205 1789636379.450421 trunc from 0x02cbf7 to 0x03b1a5 (0xe5af bytes) 000206 1789636379.457186 write 0x018f7d thru 0x024307 (0xb38b bytes) 000207 1789636379.458551 read 0x0214cf thru 0x024c4b (0x377d bytes) 000209 1789636379.459925 trunc from 0x040000 to 0x022690 (0x1d96f bytes) 000210 1789636379.478470 mapread 0x0138e3 thru 0x01c462 (0x8b80 bytes) 000211 1789636379.479430 read 0x019e58 thru 0x01de94 (0x403d bytes) 000212 1789636379.479944 trunc from 0x004ec2 to 0x0081c8 (0x3307 bytes) 000213 1789636379.486833 trunc from 0x013cf6 to 0x017a84 (0x3d8f bytes) 000214 1789636379.500319 mapwrite 0x01dd18 thru 0x02c068 (0xe351 bytes) 000215 1789636379.570702 mapread 0x00db1a thru 0x01a145 (0xc62c bytes) 000216 1789636379.572007 mapread 0x0060e0 thru 0x0096b2 (0x35d3 bytes) 000217 1789636379.573542 punch from 0x022497 to 0x0297ec (0x7356 bytes) 000219 1789636379.580420 trunc from 0x02c069 to 0x032d30 (0x6cc8 bytes) 000220 1789636379.596993 mapwrite 0x03989f thru 0x03eb1e (0x5280 bytes) 000221 1789636379.656678 mapread 0x026fa9 thru 0x0328da (0xb932 bytes) 000222 1789636379.696364 mapwrite 0x013b4a thru 0x01df94 (0xa44b bytes) 000223 1789636379.728304 mapread 0x039d7c thru 0x03eb1e (0x4da3 bytes) 000224 1789636379.733222 trunc from 0x03eb1f to 0x035821 (0x92fd bytes) 000225 1789636379.758882 write 0x03e485 thru 0x03ffff (0x1b7b bytes) 000226 1789636379.761684 mapwrite 0x03e44d thru 0x03ffff (0x1bb3 bytes) 000228 1789636379.778906 trunc from 0x040000 to 0x0322a9 (0xdd56 bytes) 000229 1789636379.815274 trunc from 0x00540f to 0x00891d (0x350f bytes) 000230 1789636379.826125 MIRROR_EXTEND to 2 mirrors 000231 1789636380.530269 fallocate 0x024336 thru 0x02cffd (0x8cc8 bytes) 000233 1789636380.564870 read 0x0049aa thru 0x008ae8 (0x413f bytes) 000234 1789636380.573114 write 0x00c5b8 thru 0x0189ff (0xc448 bytes) 000235 1789636380.588725 trunc from 0x020eef to 0x02396f (0x2a81 bytes) 000236 1789636380.594224 mapwrite 0x00d933 thru 0x014cd2 (0x73a0 bytes) 000237 1789636380.638289 trunc from 0x0322aa to 0x00f442 (0x22e67 bytes) 000238 1789636380.684732 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xf443): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf443): Mirror 2: 0xb8f42cbb (stale) CRC-32 checksum value for chunk [0, 0xf443): Mirror 1: 0x54d59522 Mirror 3: 0x54d59522 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf443: skipped 000239 1789636381.518918 mapwrite 0x00384b thru 0x009fd2 (0x6788 bytes) 000240 1789636381.658983 trunc from 0x00b11c to 0x00dc0e (0x2af3 bytes) 000241 1789636381.671405 write 0x02d4b3 thru 0x034f58 (0x7aa6 bytes) 000242 1789636381.692613 read 0x00f574 thru 0x012e12 (0x389f bytes) 000243 1789636381.710030 punch from 0x027be3 to 0x031085 (0x94a3 bytes) 000244 1789636381.739615 write 0x02ccf0 thru 0x032cd3 (0x5fe4 bytes) 000245 1789636381.751290 fallocate 0x00edd7 thru 0x0116a2 (0x28cc bytes) 000246 1789636381.767212 read 0x02596d thru 0x02e384 (0x8a18 bytes) 000247 1789636381.796612 mapwrite 0x0026f5 thru 0x003370 (0xc7c bytes) 000249 1789636381.833926 write 0x0124d1 thru 0x013eb4 (0x19e4 bytes) 000250 1789636381.839071 mapwrite 0x005dd6 thru 0x013bb3 (0xddde bytes) 000251 1789636381.890818 trunc from 0x007fb4 to 0x00d8ef (0x593c bytes) 000252 1789636381.913933 MIRROR_RESYNC 3 mirrors 000253 1789636382.646181 mapread 0x006143 thru 0x010b46 (0xaa04 bytes) 000254 1789636382.663841 write 0x020534 thru 0x023de5 (0x38b2 bytes) 000255 1789636382.710903 punch from 0x028b06 to 0x034f58 (0xc453 bytes) 000256 1789636382.717491 trunc from 0x00a453 to 0x00d8db (0x3489 bytes) 000258 1789636382.724706 mapwrite 0x001f3a thru 0x01087c (0xe943 bytes) 000259 1789636382.794241 trunc from 0x02e3f1 to 0x034f58 (0x6b68 bytes) 000260 1789636382.809260 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34f59): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f59): Mirror 2: 0xeb1e9344 (stale) Mirror 3: 0xeb1e9344 (stale) CRC-32 checksum value for chunk [0, 0x34f59): Mirror 1: 0x1c90cd5 Mirror 4: 0x1c90cd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f59: skipped 000261 1789636383.748067 punch from 0x02236b to 0x0277bc (0x5452 bytes) 000262 1789636383.784998 trunc from 0x024ed1 to 0x02914f (0x427f bytes) 000263 1789636383.792694 trunc from 0x0233ad to 0x027f39 (0x4b8d bytes) 000264 1789636383.799692 trunc from 0x034f59 to 0x01e817 (0x16741 bytes) 000265 1789636383.842808 write 0x0080e0 thru 0x00c21c (0x413d bytes) 000266 1789636383.858771 trunc from 0x01e818 to 0x039d43 (0x1b52c bytes) 000267 1789636383.873547 fallocate 0x010c76 thru 0x016a2a (0x5db5 bytes) 000268 1789636383.880937 write 0x0124bd thru 0x01665d (0x41a1 bytes) 000269 1789636383.901791 mapwrite 0x030123 thru 0x03ff58 (0xfe36 bytes) 000271 1789636383.998562 trunc from 0x03ff59 to 0x0007ec (0x3f76c bytes) 000272 1789636384.017769 punch from 0x00011a to 0x0007ec (0x6d3 bytes) 000273 1789636384.022912 mapwrite 0x025a3f thru 0x027640 (0x1c02 bytes) 000275 1789636384.070384 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x27641): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27641): Mirror 2: 0x90c991cb (stale) Mirror 3: 0x90c991cb (stale) Mirror 4: 0x967c6fb0 (stale) CRC-32 checksum value for chunk [0, 0x27641): Mirror 1: 0xc429296 Mirror 5: 0xc429296 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27641: skipped 000276 1789636385.030357 punch from 0x00159e to 0x00fe6f (0xe8d2 bytes) 000278 1789636385.094954 write 0x025147 thru 0x02aab0 (0x596a bytes) 000279 1789636385.106709 mapwrite 0x03fb12 thru 0x03ffff (0x4ee bytes) 000280 1789636385.170719 trunc from 0x009c7b to 0x0160a1 (0xc427 bytes) 000281 1789636385.181743 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb30d9519 (stale) Mirror 3: 0xb30d9519 (stale) Mirror 4: 0x46232ba2 (stale) Mirror 5: 0x125b93b1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaf837e02 Mirror 6: 0xaf837e02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000283 1789636386.118213 punch from 0x0369b4 to 0x03ffff (0x964c bytes) 000284 1789636386.163684 fallocate 0x0375d4 thru 0x03ffff (0x8a2c bytes) 000285 1789636386.174984 fallocate 0x01c61e thru 0x01ee2e (0x2811 bytes) 000286 1789636386.187828 write 0x03b5c7 thru 0x03c4bb (0xef5 bytes) 000287 1789636386.213804 mapread 0x002a2d thru 0x006e27 (0x43fb bytes) 000288 1789636386.265272 trunc from 0x03dc22 to 0x03ef7d (0x135c bytes) 000289 1789636386.274736 mapread 0x021771 thru 0x02ce45 (0xb6d5 bytes) 000290 1789636386.325253 read 0x032d65 thru 0x034183 (0x141f bytes) 000291 1789636386.330822 mapread 0x007557 thru 0x00b44d (0x3ef7 bytes) 000292 1789636386.336317 MIRROR_RESYNC 6 mirrors 000293 1789636387.938874 mapwrite 0x0178ad thru 0x01f3d7 (0x7b2b bytes) 000294 1789636388.030459 mapwrite 0x02c090 thru 0x037506 (0xb477 bytes) 000295 1789636388.062686 trunc from 0x020ffa to 0x02e733 (0xd73a bytes) 000296 1789636388.074419 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x91af3866 (stale) Mirror 4: 0x91af3866 (stale) Mirror 5: 0x91af3866 (stale) Mirror 6: 0x91af3866 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x91af3866 (stale) Mirror 4: 0x91af3866 (stale) Mirror 5: 0x91af3866 (stale) Mirror 6: 0x91af3866 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000297 1789636388.393807 fallocate 0x0156e3 thru 0x01ceba (0x77d8 bytes) 000298 1789636388.398522 punch from 0x015c0f to 0x02194b (0xbd3d bytes) 000299 1789636388.408453 mapread 0x03e880 thru 0x03ffff (0x1780 bytes) 000300 1789636388.418854 read 0x000378 thru 0x0077ec (0x7475 bytes) 000301 1789636388.425510 trunc from 0x040000 to 0x02a0e6 (0x15f19 bytes) 000302 1789636388.442609 mapread 0x021da0 thru 0x023a4a (0x1cab bytes) 000303 1789636388.448945 mapwrite 0x0324c3 thru 0x03d6c3 (0xb201 bytes) 000304 1789636388.479767 write 0x03994a thru 0x03ffff (0x66b6 bytes) 000306 1789636388.482231 punch from 0x0175d4 to 0x017f74 (0x9a1 bytes) 000307 1789636388.486818 read 0x024a6e thru 0x025ca3 (0x1236 bytes) 000308 1789636388.487581 trunc from 0x03403c to 0x03ffff (0xbfc4 bytes) 000309 1789636388.500259 trunc from 0x008853 to 0x01815f (0xf90d bytes) 000310 1789636388.505519 fallocate 0x00a9e8 thru 0x012e5d (0x8476 bytes) 000311 1789636388.510446 mapwrite 0x036472 thru 0x03ffff (0x9b8e bytes) 000313 1789636388.530315 write 0x02b3f7 thru 0x0312a3 (0x5ead bytes) 000314 1789636388.531279 write 0x021218 thru 0x0214c3 (0x2ac bytes) 000315 1789636388.532079 trunc from 0x012ada to 0x018c51 (0x6178 bytes) 000316 1789636388.536994 punch from 0x007201 to 0x00e0e5 (0x6ee5 bytes) 000317 1789636388.543648 trunc from 0x040000 to 0x000d79 (0x3f286 bytes) 000319 1789636388.570850 fallocate 0x034cc9 thru 0x03c3b7 (0x76ef bytes) 000320 1789636388.576127 fallocate 0x007b4c thru 0x015cd4 (0xe189 bytes) 000321 1789636388.584253 trunc from 0x000d7a to 0x005321 (0x45a8 bytes) 000322 1789636388.604554 mapwrite 0x038c46 thru 0x03ffff (0x73ba bytes) 000323 1789636388.672506 mapwrite 0x02bce5 thru 0x034b05 (0x8e21 bytes) 000324 1789636388.773469 punch from 0x0371ae to 0x03ffff (0x8e52 bytes) 000325 1789636388.782133 trunc from 0x040000 to 0x00330f (0x3ccf0 bytes) 000326 1789636388.811514 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3310): Mirror 4: 0x13c8aed5 (stale) Mirror 5: 0x13c8aed5 (stale) Mirror 6: 0x13c8aed5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3310): Mirror 4: 0x13c8aed5 (stale) Mirror 5: 0x13c8aed5 (stale) Mirror 6: 0x13c8aed5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3310: skipped 000327 1789636389.060957 trunc from 0x0021d2 to 0x00330f (0x113e bytes) 000328 1789636389.065629 punch from 0x000cae to 0x00330f (0x2662 bytes) 000329 1789636389.071277 read 0x002a9e thru 0x00330f (0x872 bytes) 000330 1789636389.080074 mapread 0x000deb thru 0x00330f (0x2525 bytes) 000331 1789636389.092492 MIRROR_RESYNC 4 mirrors 000333 1789636389.902522 fallocate 0x016930 thru 0x017b45 (0x1216 bytes) 000334 1789636389.936545 trunc from 0x003310 to 0x01a5cc (0x172bd bytes) 000335 1789636389.954970 write 0x01a066 thru 0x02188a (0x7825 bytes) 000336 1789636389.959937 write 0x0040a5 thru 0x00bddf (0x7d3b bytes) 000338 1789636389.968693 trunc from 0x02188b to 0x007378 (0x1a512 bytes) 000339 1789636389.983303 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x7379): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7379): Mirror 4: 0xaba11958 (stale) Mirror 5: 0xaba11958 (stale) Mirror 6: 0xaba11958 (stale) CRC-32 checksum value for chunk [0, 0x7379): Mirror 1: 0xd3070e1a Mirror 7: 0xd3070e1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7379: skipped 000340 1789636390.827000 mapread 0x0008ba thru 0x007378 (0x6abf bytes) 000341 1789636390.848008 fallocate 0x025ce2 thru 0x02d822 (0x7b41 bytes) 000342 1789636390.902537 trunc from 0x006367 to 0x007378 (0x1012 bytes) 000343 1789636390.907314 mapread 0x0034e1 thru 0x006f87 (0x3aa7 bytes) 000344 1789636390.930676 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7379): Mirror 4: 0xaba11958 (stale) Mirror 5: 0xaba11958 (stale) Mirror 6: 0xaba11958 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7379): Mirror 4: 0xaba11958 (stale) Mirror 5: 0xaba11958 (stale) Mirror 6: 0xaba11958 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7379: skipped 000345 1789636391.356238 MIRROR_RESYNC 4 mirrors 000346 1789636392.156782 trunc from 0x007379 to 0x00547c (0x1efc bytes) 000347 1789636392.207418 trunc from 0x00547d to 0x029dec (0x24970 bytes) 000348 1789636392.229936 mapread 0x017827 thru 0x02577c (0xdf56 bytes) 000349 1789636392.282453 punch from 0x022049 to 0x029dec (0x7da4 bytes) 000350 1789636392.288587 punch from 0x010bd7 to 0x01e158 (0xd582 bytes) 000351 1789636392.300754 mapread 0x009880 thru 0x009d3d (0x4be bytes) 000352 1789636392.326371 mapread 0x01ac9f thru 0x022ffe (0x8360 bytes) 000353 1789636392.346509 MIRROR_RESYNC 4 mirrors 000354 1789636393.211580 mapread 0x023b46 thru 0x029dec (0x62a7 bytes) 000355 1789636393.236759 read 0x00f6f0 thru 0x014a76 (0x5387 bytes) 000358 1789636393.249594 read 0x010496 thru 0x013686 (0x31f1 bytes) 000359 1789636393.250469 trunc from 0x0080b1 to 0x016732 (0xe682 bytes) 000360 1789636393.294848 read 0x024070 thru 0x029dec (0x5d7d bytes) 000361 1789636393.308115 mapread 0x01cd0d thru 0x020009 (0x32fd bytes) 000362 1789636393.310770 punch from 0x00e6dd to 0x00f545 (0xe69 bytes) 000363 1789636393.318177 mapread 0x027dff thru 0x029dc5 (0x1fc7 bytes) 000364 1789636393.321778 trunc from 0x029ded to 0x01d52b (0xc8c1 bytes) 000365 1789636393.365341 read 0x016872 thru 0x01d52b (0x6cba bytes) 000366 1789636393.366125 mapread 0x0176e5 thru 0x019ba2 (0x24be bytes) 000367 1789636393.367521 read 0x009c44 thru 0x012206 (0x85c3 bytes) 000368 1789636393.378435 trunc from 0x01d52c to 0x029fef (0xcac4 bytes) 000369 1789636393.400904 mapread 0x001326 thru 0x00752b (0x6206 bytes) 000370 1789636393.401941 mapread 0x005718 thru 0x01036c (0xac55 bytes) 000372 1789636393.403954 mapread 0x012b6c thru 0x017579 (0x4a0e bytes) 000373 1789636393.404922 write 0x001a19 thru 0x002155 (0x73d bytes) 000374 1789636393.407095 trunc from 0x013b80 to 0x014e40 (0x12c1 bytes) 000375 1789636393.413303 write 0x038e5d thru 0x03ffff (0x71a3 bytes) 000376 1789636393.414766 mapread 0x03c835 thru 0x03ffff (0x37cb bytes) 000377 1789636393.416557 read 0x03b543 thru 0x03ffff (0x4abd bytes) 000378 1789636393.417814 trunc from 0x0028ff to 0x006b58 (0x425a bytes) 000380 1789636393.432494 punch from 0x007542 to 0x0097ca (0x2289 bytes) 000381 1789636393.438690 trunc from 0x040000 to 0x02803c (0x17fc3 bytes) 000382 1789636393.457210 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2803d): Mirror 5: 0x922582e2 (stale) Mirror 6: 0x922582e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2803d): Mirror 5: 0x922582e2 (stale) Mirror 6: 0x922582e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2803d: skipped 000384 1789636393.691828 punch from 0x00b3cc to 0x010c8f (0x58c4 bytes) 000385 1789636393.697729 trunc from 0x0265c1 to 0x02803c (0x1a7c bytes) 000386 1789636393.705031 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2803d): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2803d): Mirror 5: 0x922582e2 (stale) Mirror 6: 0x922582e2 (stale) CRC-32 checksum value for chunk [0, 0x2803d): Mirror 1: 0x1b91d91 Mirror 7: 0x1b91d91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2803d: skipped 000387 1789636394.261405 mapwrite 0x02186f thru 0x027e89 (0x661b bytes) 000389 1789636394.365692 punch from 0x007d04 to 0x00c80b (0x4b08 bytes) 000391 1789636394.373433 MIRROR_RESYNC 4 mirrors 000392 1789636394.988853 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000393 1789636395.283403 punch from 0x002672 to 0x005a08 (0x3397 bytes) 000394 1789636395.326181 punch from 0x010341 to 0x012ec5 (0x2b85 bytes) 000395 1789636395.330701 trunc from 0x02803d to 0x035d99 (0xdd5d bytes) 000396 1789636395.361561 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35d9a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35d9a): Mirror 6: 0xc4ab16a7 (stale) Mirror 7: 0xc4ab16a7 (stale) CRC-32 checksum value for chunk [0, 0x35d9a): Mirror 1: 0xc4ab16a7 Mirror 8: 0xc4ab16a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35d9a: skipped 000397 1789636396.071333 mapread 0x022fb2 thru 0x02ec5c (0xbcab bytes) 000398 1789636396.119558 read 0x02bfb2 thru 0x02dc83 (0x1cd2 bytes) 000399 1789636396.126112 punch from 0x00bb45 to 0x01ae1b (0xf2d7 bytes) 000400 1789636396.169957 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35d9a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35d9a): Mirror 6: 0xc4ab16a7 (stale) Mirror 7: 0xc4ab16a7 (stale) Mirror 8: 0xc4ab16a7 (stale) CRC-32 checksum value for chunk [0, 0x35d9a): Mirror 1: 0xc4ab16a7 Mirror 9: 0xc4ab16a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35d9a: skipped 000402 1789636396.891388 write 0x01240b thru 0x01eddf (0xc9d5 bytes) 000403 1789636396.941663 read 0x0155f1 thru 0x0226d4 (0xd0e4 bytes) 000404 1789636396.953707 mapwrite 0x03433e thru 0x03ae08 (0x6acb bytes) 000405 1789636397.019302 fallocate 0x021e64 thru 0x026e11 (0x4fae bytes) 000406 1789636397.024089 mapread 0x01daac thru 0x029456 (0xb9ab bytes) 000407 1789636397.034930 punch from 0x02c254 to 0x032a78 (0x6825 bytes) 000408 1789636397.047572 trunc from 0x03a00d to 0x03ae08 (0xdfc bytes) 000409 1789636397.060721 MIRROR_RESYNC 5 mirrors 000410 1789636398.258991 read 0x029ec5 thru 0x0369ad (0xcae9 bytes) 000411 1789636398.272582 trunc from 0x03ae09 to 0x014ab7 (0x26351 bytes) 000412 1789636398.376917 trunc from 0x014ab8 to 0x02f160 (0x1a6a9 bytes) 000413 1789636398.396607 write 0x03239b thru 0x03cdc8 (0xaa2e bytes) 000414 1789636398.398208 mapread 0x002dff thru 0x00b851 (0x8a53 bytes) 000415 1789636398.422495 trunc from 0x03cdc9 to 0x03994e (0x347a bytes) 000416 1789636398.436789 fallocate 0x00c330 thru 0x01a3c7 (0xe098 bytes) 000417 1789636398.440237 trunc from 0x002cd2 to 0x011158 (0xe487 bytes) 000418 1789636398.445588 trunc from 0x03994f to 0x039814 (0x13a bytes) 000419 1789636398.462783 write 0x03ffc7 thru 0x03ffff (0x39 bytes) 000421 1789636398.464210 trunc from 0x040000 to 0x01227f (0x2dd80 bytes) 000422 1789636398.498725 mapread 0x01167a thru 0x01227f (0xc06 bytes) 000423 1789636398.514554 trunc from 0x00ba6b to 0x00ee69 (0x33ff bytes) 000424 1789636398.524525 write 0x00675e thru 0x015c33 (0xf4d6 bytes) 000425 1789636398.536017 trunc from 0x00357d to 0x008f14 (0x5998 bytes) 000426 1789636398.577298 read 0x0042f6 thru 0x00a2a4 (0x5faf bytes) 000427 1789636398.588663 fallocate 0x036f7d thru 0x03ffff (0x9083 bytes) 000428 1789636398.598386 read 0x003adf thru 0x00d6aa (0x9bcc bytes) 000429 1789636398.599275 punch from 0x006bba to 0x00f0cd (0x8514 bytes) 000430 1789636398.615025 mapwrite 0x00cbf1 thru 0x0164f2 (0x9902 bytes) 000431 1789636398.657638 fallocate 0x03e4e7 thru 0x03ffff (0x1b19 bytes) 000432 1789636398.664452 mapread 0x00d0b9 thru 0x0136fa (0x6642 bytes) 000433 1789636398.665172 mapread 0x00f423 thru 0x0164f2 (0x70d0 bytes) 000434 1789636398.665955 fallocate 0x03b2fa thru 0x03c83b (0x1542 bytes) 000435 1789636398.671991 trunc from 0x0164f3 to 0x01942d (0x2f3b bytes) 000436 1789636398.693010 write 0x022a1b thru 0x02311b (0x701 bytes) 000437 1789636398.694191 trunc from 0x02311c to 0x03dcf0 (0x1abd5 bytes) 000438 1789636398.709394 read 0x023537 thru 0x0274be (0x3f88 bytes) 000439 1789636398.715870 trunc from 0x03dcf1 to 0x02757b (0x16775 bytes) 000440 1789636398.731529 mapread 0x0244f7 thru 0x02757b (0x3085 bytes) 000441 1789636398.732803 mapwrite 0x017874 thru 0x01dddb (0x6568 bytes) 000442 1789636398.751905 write 0x038570 thru 0x03ffff (0x7a90 bytes) 000443 1789636398.753237 trunc from 0x007c57 to 0x00cacc (0x4e76 bytes) 000445 1789636398.766818 fallocate 0x015a6b thru 0x018ad3 (0x3069 bytes) 000446 1789636398.772094 trunc from 0x040000 to 0x017294 (0x28d6b bytes) 000447 1789636398.792146 punch from 0x00e29b to 0x017294 (0x8ffa bytes) 000448 1789636398.797502 read 0x010a1e thru 0x0112d0 (0x8b3 bytes) 000449 1789636398.810468 trunc from 0x01661c to 0x017294 (0xc79 bytes) 000450 1789636398.815922 fallocate 0x0080e9 thru 0x00811c (0x34 bytes) 000451 1789636398.819517 trunc from 0x017295 to 0x037859 (0x205c5 bytes) 000452 1789636398.836583 mapwrite 0x03a17f thru 0x03ffff (0x5e81 bytes) 000453 1789636398.885428 punch from 0x0279ca to 0x036ef8 (0xf52f bytes) 000454 1789636398.891436 punch from 0x017439 to 0x01b3f2 (0x3fba bytes) 000455 1789636398.896363 fallocate 0x038cc7 thru 0x03c9ae (0x3ce8 bytes) 000456 1789636398.901786 trunc from 0x040000 to 0x019df2 (0x2620d bytes) 000457 1789636398.917665 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19df3): Mirror 7: 0x8ad598c7 (stale) Mirror 8: 0x8ad598c7 (stale) Mirror 9: 0x8ad598c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19df3): Mirror 7: 0x8ad598c7 (stale) Mirror 8: 0x8ad598c7 (stale) Mirror 9: 0x8ad598c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19df3: skipped 000458 1789636399.172823 mapread 0x00efe1 thru 0x01762d (0x864d bytes) 000459 1789636399.188976 MIRROR_RESYNC 4 mirrors 000460 1789636400.095469 trunc from 0x007bf5 to 0x00817f (0x58b bytes) 000461 1789636400.138336 mapread 0x017213 thru 0x019df2 (0x2be0 bytes) 000462 1789636400.166969 punch from 0x01785c to 0x019df2 (0x2597 bytes) 000463 1789636400.174970 punch from 0x01875c to 0x019df2 (0x1697 bytes) 000464 1789636400.181140 trunc from 0x015b99 to 0x017df3 (0x225b bytes) 000465 1789636400.188750 mapread 0x00acfe thru 0x015f16 (0xb219 bytes) 000466 1789636400.206938 trunc from 0x019df3 to 0x035e0e (0x1c01c bytes) 000467 1789636400.225199 punch from 0x0190ab to 0x028072 (0xefc8 bytes) 000468 1789636400.232441 trunc from 0x01c66c to 0x024122 (0x7ab7 bytes) 000469 1789636400.237433 trunc from 0x0204c1 to 0x022b0b (0x264b bytes) 000471 1789636400.245179 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35e0f): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e0f): Mirror 7: 0xcbb1c455 (stale) Mirror 8: 0xcbb1c455 (stale) Mirror 9: 0xcbb1c455 (stale) CRC-32 checksum value for chunk [0, 0x35e0f): Mirror 1: 0xcbb1c455 Mirror 10: 0xcbb1c455 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e0f: skipped 000472 1789636401.088349 write 0x03839a thru 0x03ffff (0x7c66 bytes) 000473 1789636401.132009 trunc from 0x025892 to 0x02e4fc (0x8c6b bytes) 000474 1789636401.143709 trunc from 0x040000 to 0x01a04a (0x25fb5 bytes) 000475 1789636401.163955 punch from 0x0114d5 to 0x01a04a (0x8b76 bytes) 000476 1789636401.180364 MIRROR_RESYNC 5 mirrors 000477 1789636402.275436 read 0x0055f8 thru 0x014ae2 (0xf4eb bytes) 000478 1789636402.302430 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000479 1789636402.645113 trunc from 0x007f7b to 0x00e38c (0x6412 bytes) 000480 1789636402.687380 mapread 0x01032c thru 0x012c60 (0x2935 bytes) 000481 1789636402.705636 trunc from 0x01644e to 0x017d8f (0x1942 bytes) 000482 1789636402.710474 write 0x029660 thru 0x037acc (0xe46d bytes) 000483 1789636402.715763 write 0x01f8dc thru 0x01fbed (0x312 bytes) 000484 1789636402.721946 read 0x01c299 thru 0x02c1f1 (0xff59 bytes) 000485 1789636402.731657 punch from 0x01e826 to 0x02a971 (0xc14c bytes) 000486 1789636402.750176 trunc from 0x02035b to 0x0288ba (0x8560 bytes) 000487 1789636402.756836 mapread 0x0283e1 thru 0x0359c5 (0xd5e5 bytes) 000488 1789636402.767081 punch from 0x02c09d to 0x032209 (0x616d bytes) 000489 1789636402.771176 trunc from 0x037acd to 0x03d5f2 (0x5b26 bytes) 000490 1789636402.788720 punch from 0x01c1c2 to 0x023ab7 (0x78f6 bytes) 000491 1789636402.793833 mapwrite 0x000fdf thru 0x00adac (0x9dce bytes) 000492 1789636402.825202 MIRROR_RESYNC 4 mirrors 000493 1789636404.051792 trunc from 0x015d54 to 0x020e09 (0xb0b6 bytes) 000494 1789636404.092026 fallocate 0x005256 thru 0x00f39d (0xa148 bytes) 000495 1789636404.096342 punch from 0x00142b to 0x009e51 (0x8a27 bytes) 000496 1789636404.100259 trunc from 0x03d5f3 to 0x02acd8 (0x1291a bytes) 000497 1789636404.115067 trunc from 0x017066 to 0x023fed (0xcf88 bytes) 000498 1789636404.118980 trunc from 0x0115e5 to 0x01cb5a (0xb576 bytes) 000499 1789636404.122504 trunc from 0x02acd9 to 0x03144f (0x6777 bytes) 000500 1789636404.137454 punch from 0x01c86a to 0x029e68 (0xd5ff bytes) 000502 1789636404.147890 fallocate 0x0097a4 thru 0x010894 (0x70f1 bytes) 000503 1789636404.152231 trunc from 0x031450 to 0x002955 (0x2eafa bytes) 000504 1789636404.170028 punch from 0x00142f to 0x0021af (0xd81 bytes) 000505 1789636404.176472 mapread 0x0025f6 thru 0x002955 (0x360 bytes) 000506 1789636404.181685 trunc from 0x002078 to 0x002955 (0x8de bytes) 000507 1789636404.186542 fallocate 0x00d719 thru 0x01b4d5 (0xddbd bytes) 000508 1789636404.191807 read 0x00f2c1 thru 0x0189a7 (0x96e7 bytes) 000509 1789636404.206921 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b4d6): Mirror 7: 0x43c9d050 (stale) Mirror 10: 0x43c9d050 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b4d6): Mirror 7: 0x43c9d050 (stale) Mirror 10: 0x43c9d050 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b4d6: skipped 000510 1789636404.494397 fallocate 0x000c5e thru 0x00ce81 (0xc224 bytes) 000511 1789636404.504792 punch from 0x001e40 to 0x010dd5 (0xef96 bytes) 000512 1789636404.509862 trunc from 0x003ee9 to 0x013c22 (0xfd3a bytes) 000513 1789636404.517483 mapread 0x01753f thru 0x01b4d5 (0x3f97 bytes) 000514 1789636404.549811 MIRROR_RESYNC 3 mirrors 000515 1789636405.140416 MIRROR_RESYNC 3 mirrors 000516 1789636405.489408 trunc from 0x01b4d6 to 0x01e8e8 (0x3413 bytes) 000517 1789636405.552392 trunc from 0x01d221 to 0x01d6f7 (0x4d7 bytes) 000518 1789636405.558074 punch from 0x00c88b to 0x017cdf (0xb455 bytes) 000519 1789636405.563447 mapwrite 0x01125f thru 0x01c0a9 (0xae4b bytes) 000520 1789636405.612104 fallocate 0x011ec1 thru 0x01f412 (0xd552 bytes) 000522 1789636405.621532 read 0x007a0b thru 0x011501 (0x9af7 bytes) 000523 1789636405.630233 punch from 0x016c09 to 0x01f412 (0x880a bytes) 000524 1789636405.648575 read 0x019823 thru 0x01f412 (0x5bf0 bytes) 000525 1789636405.665257 fallocate 0x03e74a thru 0x03ffff (0x18b6 bytes) 000526 1789636405.671344 fallocate 0x00902b thru 0x016625 (0xd5fb bytes) 000527 1789636405.676754 trunc from 0x040000 to 0x02ed8a (0x11275 bytes) 000528 1789636405.697316 write 0x01cd6b thru 0x029c2f (0xcec5 bytes) 000529 1789636405.705563 read 0x022102 thru 0x02b98d (0x988c bytes) 000531 1789636405.718172 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed8b): Mirror 10: 0x7773c4d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed8b): Mirror 10: 0x7773c4d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed8b: skipped 000532 1789636405.986569 write 0x00a87b thru 0x0115bd (0x6d43 bytes) 000533 1789636406.004554 punch from 0x01f3c5 to 0x02a2fd (0xaf39 bytes) 000534 1789636406.013814 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ed8b): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed8b): Mirror 10: 0x7773c4d1 (stale) CRC-32 checksum value for chunk [0, 0x2ed8b): Mirror 1: 0x21cb7928 Mirror 11: 0x21cb7928 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed8b: skipped 000535 1789636406.907718 write 0x00f920 thru 0x011840 (0x1f21 bytes) 000536 1789636406.959793 fallocate 0x01acf3 thru 0x027319 (0xc627 bytes) 000537 1789636406.975775 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed8b): Mirror 10: 0x7773c4d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed8b): Mirror 10: 0x7773c4d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed8b: skipped 000538 1789636407.213030 punch from 0x001a88 to 0x00faea (0xe063 bytes) 000539 1789636407.221289 fallocate 0x017e38 thru 0x01ac54 (0x2e1d bytes) 000540 1789636407.228474 mapwrite 0x03c7d2 thru 0x03ffff (0x382e bytes) 000541 1789636407.292082 fallocate 0x02deef thru 0x037de0 (0x9ef2 bytes) 000543 1789636407.299959 punch from 0x00f602 to 0x0198ec (0xa2eb bytes) 000544 1789636407.303949 write 0x007e94 thru 0x0160f3 (0xe260 bytes) 000545 1789636407.311137 mapread 0x019431 thru 0x021353 (0x7f23 bytes) 000546 1789636407.316505 trunc from 0x008e6a to 0x017de2 (0xef79 bytes) 000548 1789636407.327188 trunc from 0x040000 to 0x02a7a3 (0x1585c bytes) 000549 1789636407.340890 write 0x03c2e6 thru 0x03ffff (0x3d1a bytes) 000550 1789636407.342224 fallocate 0x034b1b thru 0x03c7f1 (0x7cd7 bytes) 000551 1789636407.346713 read 0x00367f thru 0x011c76 (0xe5f8 bytes) 000552 1789636407.365969 read 0x03584e thru 0x03eb89 (0x933c bytes) 000554 1789636407.378026 punch from 0x02b3e5 to 0x03b1fa (0xfe16 bytes) 000556 1789636407.383631 fallocate 0x03a20c thru 0x03ffff (0x5df4 bytes) 000557 1789636407.389867 read 0x01cbf1 thru 0x028122 (0xb532 bytes) 000558 1789636407.390523 fallocate 0x0283e3 thru 0x030fd1 (0x8bef bytes) 000560 1789636407.395900 trunc from 0x040000 to 0x014212 (0x2bded bytes) 000561 1789636407.406888 mapwrite 0x00ab7c thru 0x0114a5 (0x692a bytes) 000562 1789636407.420667 mapwrite 0x008918 thru 0x00c161 (0x384a bytes) 000563 1789636407.447231 trunc from 0x00bce9 to 0x014212 (0x852a bytes) 000564 1789636407.455400 trunc from 0x00cf4b to 0x014212 (0x72c8 bytes) 000565 1789636407.460811 punch from 0x00eff8 to 0x014212 (0x521b bytes) 000566 1789636407.471441 trunc from 0x00578d to 0x00e636 (0x8eaa bytes) 000567 1789636407.480637 trunc from 0x008d65 to 0x00b2b3 (0x254f bytes) 000568 1789636407.487593 mapread 0x008c5c thru 0x014212 (0xb5b7 bytes) 000569 1789636407.505393 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x14213): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14213): Mirror 10: 0xc2c46370 (stale) CRC-32 checksum value for chunk [0, 0x14213): Mirror 1: 0xc2c46370 Mirror 11: 0xc2c46370 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14213: skipped 000570 1789636408.213626 write 0x032dd9 thru 0x039365 (0x658d bytes) 000571 1789636408.273537 read 0x006572 thru 0x0159b5 (0xf444 bytes) 000572 1789636408.286308 mapread 0x0200c8 thru 0x020bcd (0xb06 bytes) 000573 1789636408.294198 read 0x02823d thru 0x02d642 (0x5406 bytes) 000574 1789636408.304745 trunc from 0x039366 to 0x029ad8 (0xf88d bytes) 000575 1789636408.345822 trunc from 0x0068f6 to 0x013283 (0xc98e bytes) 000576 1789636408.354622 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29ad9): Mirror 11: 0x2d8dc1a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ad9): Mirror 11: 0x2d8dc1a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ad9: skipped 000577 1789636408.641748 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x29ad9): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ad9): Mirror 11: 0x2d8dc1a1 (stale) CRC-32 checksum value for chunk [0, 0x29ad9): Mirror 1: 0x2d8dc1a1 Mirror 12: 0x2d8dc1a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ad9: skipped 000578 1789636409.499738 trunc from 0x0120bf to 0x0199cd (0x790f bytes) 000579 1789636409.545455 write 0x02c310 thru 0x02d774 (0x1465 bytes) 000580 1789636409.554578 mapread 0x00fe3b thru 0x01de0b (0xdfd1 bytes) 000581 1789636409.594288 mapwrite 0x03ec9c thru 0x03ffff (0x1364 bytes) 000582 1789636409.644574 punch from 0x00d0c2 to 0x00ee91 (0x1dd0 bytes) 000583 1789636409.650245 write 0x028e9c thru 0x02e49f (0x5604 bytes) 000584 1789636409.659976 trunc from 0x0332d6 to 0x033b22 (0x84d bytes) 000585 1789636409.663465 trunc from 0x0118c9 to 0x0170c0 (0x57f8 bytes) 000586 1789636409.672255 MIRROR_RESYNC 3 mirrors 000587 1789636410.537574 write 0x0230ab thru 0x025670 (0x25c6 bytes) 000588 1789636410.568428 mapwrite 0x018275 thru 0x027659 (0xf3e5 bytes) 000589 1789636410.611900 read 0x03cfe9 thru 0x03ffff (0x3017 bytes) 000590 1789636410.618212 fallocate 0x02c8ab thru 0x0334c1 (0x6c17 bytes) 000591 1789636410.624187 mapread 0x0221ab thru 0x0309b5 (0xe80b bytes) 000593 1789636410.637469 fallocate 0x012d00 thru 0x012e3f (0x140 bytes) 000594 1789636410.643533 read 0x03dad9 thru 0x03ffff (0x2527 bytes) 000596 1789636410.648507 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xe48f0d8a (stale) Mirror 12: 0xe48f0d8a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe8e21875 Mirror 13: 0xe8e21875 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1789636411.594400 trunc from 0x040000 to 0x02285f (0x1d7a0 bytes) 000598 1789636411.639919 fallocate 0x003799 thru 0x00d405 (0x9c6d bytes) 000599 1789636411.645691 fallocate 0x007a71 thru 0x00bde5 (0x4375 bytes) 000600 1789636411.652341 mapwrite 0x01b9b7 thru 0x01ec23 (0x326d bytes) 000601 1789636411.689933 mapread 0x00e64c thru 0x010b6e (0x2523 bytes) 000602 1789636411.700724 punch from 0x01e65e to 0x02285f (0x4202 bytes) 000603 1789636411.706442 trunc from 0x022860 to 0x016ca1 (0xbbbe bytes) 000604 1789636411.728289 trunc from 0x012f4e to 0x015b68 (0x2c1b bytes) 000605 1789636411.736312 mapread 0x007cdf thru 0x0082a7 (0x5c9 bytes) 000606 1789636411.739459 trunc from 0x016ca2 to 0x01e888 (0x7be7 bytes) 000608 1789636411.765376 trunc from 0x01d044 to 0x01d634 (0x5f1 bytes) 000609 1789636411.772945 mapwrite 0x033bda thru 0x03781a (0x3c41 bytes) 000610 1789636411.835114 trunc from 0x00e741 to 0x00f274 (0xb34 bytes) 000611 1789636411.848904 write 0x02897c thru 0x029097 (0x71c bytes) 000612 1789636411.871425 trunc from 0x03781b to 0x02facb (0x7d4f bytes) 000613 1789636411.895923 mapwrite 0x007bf4 thru 0x00947a (0x1887 bytes) 000614 1789636411.922231 trunc from 0x006f89 to 0x00de65 (0x6edd bytes) 000615 1789636411.929694 mapread 0x029a37 thru 0x02facb (0x6095 bytes) 000616 1789636411.941412 trunc from 0x01e8b6 to 0x023eb3 (0x55fe bytes) 000617 1789636411.946005 mapread 0x02efd1 thru 0x02facb (0xafb bytes) 000618 1789636411.947460 read 0x014653 thru 0x023b37 (0xf4e5 bytes) 000619 1789636411.958197 mapwrite 0x02d671 thru 0x0328ba (0x524a bytes) 000620 1789636412.006020 trunc from 0x0328bb to 0x031016 (0x18a4 bytes) 000621 1789636412.039900 write 0x02f2a3 thru 0x03b585 (0xc2e3 bytes) 000622 1789636412.041733 trunc from 0x023694 to 0x02ffe0 (0xc94d bytes) 000623 1789636412.072634 trunc from 0x03b586 to 0x039a20 (0x1b65 bytes) 000624 1789636412.105502 mapwrite 0x020075 thru 0x02cb53 (0xcadf bytes) 000625 1789636412.162971 mapread 0x00c1d6 thru 0x00fa9a (0x38c5 bytes) 000626 1789636412.180902 fallocate 0x013b1a thru 0x01a9fd (0x6ee4 bytes) 000627 1789636412.196548 mapread 0x01cb12 thru 0x01ccea (0x1d9 bytes) 000628 1789636412.197790 trunc from 0x032557 to 0x0399fb (0x74a5 bytes) 000629 1789636412.225447 punch from 0x005eea to 0x0139e7 (0xdafe bytes) 000630 1789636412.236569 trunc from 0x039a21 to 0x025140 (0x148e0 bytes) 000631 1789636412.283773 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25141): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25141): Mirror 11: 0x33ffa507 (stale) Mirror 12: 0x33ffa507 (stale) Mirror 13: 0xcf3e5c7a (stale) CRC-32 checksum value for chunk [0, 0x25141): Mirror 1: 0x9512c7e8 Mirror 14: 0x9512c7e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25141: skipped 000632 1789636413.327295 fallocate 0x007654 thru 0x014289 (0xcc36 bytes) 000634 1789636413.360784 read 0x00b1fa thru 0x01672a (0xb531 bytes) 000635 1789636413.368684 punch from 0x01ae5a to 0x01bf4f (0x10f6 bytes) 000636 1789636413.373793 trunc from 0x0094e9 to 0x00b621 (0x2139 bytes) 000637 1789636413.378770 mapwrite 0x0158e9 thru 0x01ebca (0x92e2 bytes) 000638 1789636413.409714 mapread 0x00e739 thru 0x01b94d (0xd215 bytes) 000639 1789636413.410724 punch from 0x0170df to 0x02310f (0xc031 bytes) 000640 1789636413.416889 punch from 0x024792 to 0x025140 (0x9af bytes) 000641 1789636413.422093 punch from 0x0227ee to 0x0233b9 (0xbcc bytes) 000642 1789636413.425801 mapwrite 0x0150f2 thru 0x0201d4 (0xb0e3 bytes) 000644 1789636413.472715 mapread 0x02062e thru 0x023cdf (0x36b2 bytes) 000645 1789636413.474000 trunc from 0x025141 to 0x022176 (0x2fca bytes) 000646 1789636413.502484 mapwrite 0x00c5b2 thru 0x0170d4 (0xab23 bytes) 000648 1789636413.536466 trunc from 0x022177 to 0x0214ee (0xc88 bytes) 000649 1789636413.555510 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x214ef): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x214ef): Mirror 11: 0x3283e0fa (stale) Mirror 12: 0x3283e0fa (stale) Mirror 13: 0xb1796dc3 (stale) Mirror 14: 0x20090677 (stale) CRC-32 checksum value for chunk [0, 0x214ef): Mirror 1: 0x8875419a Mirror 15: 0x8875419a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x214ef: skipped 000650 1789636414.531858 fallocate 0x03596c thru 0x03bacf (0x6164 bytes) 000651 1789636414.618463 trunc from 0x0214ef to 0x017d28 (0x97c6 bytes) 000652 1789636414.712257 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17d29): Mirror 11: 0x4efbe3d9 (stale) Mirror 12: 0x4efbe3d9 (stale) Mirror 13: 0x4efbe3d9 (stale) Mirror 14: 0x4efbe3d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17d29): Mirror 11: 0x4efbe3d9 (stale) Mirror 12: 0x4efbe3d9 (stale) Mirror 13: 0x4efbe3d9 (stale) Mirror 14: 0x4efbe3d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17d29: skipped 000653 1789636415.140997 write 0x034683 thru 0x039069 (0x49e7 bytes) 000654 1789636415.151208 read 0x03796d thru 0x039069 (0x16fd bytes) 000655 1789636415.152559 trunc from 0x01f7c7 to 0x01fbf8 (0x432 bytes) 000656 1789636415.163456 mapread 0x034fd5 thru 0x039069 (0x4095 bytes) 000657 1789636415.165700 read 0x022b22 thru 0x030136 (0xd615 bytes) 000658 1789636415.191933 trunc from 0x037b4d to 0x039069 (0x151d bytes) 000659 1789636415.230463 write 0x037949 thru 0x03ab56 (0x320e bytes) 000660 1789636415.243995 trunc from 0x008487 to 0x015690 (0xd20a bytes) 000661 1789636415.254585 punch from 0x009921 to 0x010277 (0x6957 bytes) 000662 1789636415.260987 punch from 0x002fbf to 0x0120a5 (0xf0e7 bytes) 000663 1789636415.270657 mapread 0x0158ec thru 0x019458 (0x3b6d bytes) 000664 1789636415.300859 MIRROR_RESYNC 5 mirrors 000665 1789636416.678546 MIRROR_RESYNC 5 mirrors 000667 1789636417.095813 mapwrite 0x00dc1a thru 0x01d085 (0xf46c bytes) 000668 1789636417.183681 punch from 0x028b9f to 0x02a9b9 (0x1e1b bytes) 000669 1789636417.188451 fallocate 0x0131ec thru 0x01e562 (0xb377 bytes) 000670 1789636417.193328 write 0x0018c0 thru 0x005122 (0x3863 bytes) 000672 1789636417.203199 fallocate 0x02f8a4 thru 0x036d0b (0x7468 bytes) 000673 1789636417.207623 mapread 0x00137d thru 0x00f156 (0xddda bytes) 000674 1789636417.217213 mapread 0x018be1 thru 0x02797b (0xed9b bytes) 000675 1789636417.226663 write 0x03afc2 thru 0x03da79 (0x2ab8 bytes) 000677 1789636417.233169 trunc from 0x03da7a to 0x01da17 (0x20062 bytes) 000678 1789636417.264923 MIRROR_RESYNC 5 mirrors 000679 1789636418.216730 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000680 1789636418.481393 write 0x006f33 thru 0x0134df (0xc5ad bytes) 000681 1789636418.529262 trunc from 0x01da18 to 0x02bf67 (0xe550 bytes) 000682 1789636418.559291 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bf68): Mirror 12: 0x8c5ef8d7 (stale) Mirror 13: 0x8c5ef8d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf68): Mirror 12: 0x8c5ef8d7 (stale) Mirror 13: 0x8c5ef8d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf68: skipped 000683 1789636418.872690 MIRROR_RESYNC 3 mirrors 000684 1789636419.495207 mapwrite 0x01c550 thru 0x023e31 (0x78e2 bytes) 000685 1789636419.575868 punch from 0x008239 to 0x015299 (0xd061 bytes) 000686 1789636419.580911 fallocate 0x02bedb thru 0x03474b (0x8871 bytes) 000687 1789636419.586381 mapread 0x01377a thru 0x02090f (0xd196 bytes) 000688 1789636419.598159 write 0x01e02d thru 0x0296b7 (0xb68b bytes) 000689 1789636419.607646 MIRROR_RESYNC 3 mirrors 000690 1789636420.298157 mapwrite 0x01ffb8 thru 0x023843 (0x388c bytes) 000691 1789636420.369917 punch from 0x029978 to 0x02bf67 (0x25f0 bytes) 000692 1789636420.373694 trunc from 0x01cffc to 0x0213de (0x43e3 bytes) 000693 1789636420.377190 read 0x00a731 thru 0x00b77a (0x104a bytes) 000694 1789636420.381387 trunc from 0x00f0bd to 0x0137a5 (0x46e9 bytes) 000695 1789636420.385511 fallocate 0x0315f0 thru 0x036f2c (0x593d bytes) 000696 1789636420.389366 punch from 0x00cfd3 to 0x01367f (0x66ad bytes) 000697 1789636420.393361 trunc from 0x005339 to 0x00a60f (0x52d7 bytes) 000698 1789636420.397547 write 0x03aa88 thru 0x03ffff (0x5578 bytes) 000699 1789636420.398829 write 0x0254b3 thru 0x031529 (0xc077 bytes) 000700 1789636420.403470 punch from 0x011909 to 0x01fac5 (0xe1bd bytes) 000701 1789636420.407323 mapread 0x0333a3 thru 0x03d3a3 (0xa001 bytes) 000702 1789636420.413774 mapwrite 0x021424 thru 0x021d91 (0x96e bytes) 000703 1789636420.436501 mapread 0x015700 thru 0x02434c (0xec4d bytes) 000704 1789636420.445023 mapwrite 0x025a90 thru 0x028275 (0x27e6 bytes) 000705 1789636420.455491 trunc from 0x040000 to 0x00051b (0x3fae4 bytes) 000706 1789636420.471263 read 0x000092 thru 0x00051b (0x48a bytes) 000707 1789636420.475073 trunc from 0x00051c to 0x01666d (0x16152 bytes) 000708 1789636420.487194 mapread 0x00ba7b thru 0x01486b (0x8df1 bytes) 000709 1789636420.493972 trunc from 0x00f8aa to 0x01666d (0x6dc4 bytes) 000710 1789636420.497371 fallocate 0x02c06e thru 0x038fc5 (0xcf58 bytes) 000711 1789636420.500918 read 0x0239a8 thru 0x02cc7a (0x92d3 bytes) 000712 1789636420.507543 mapread 0x0304da thru 0x031e9c (0x19c3 bytes) 000713 1789636420.508590 mapread 0x036ee2 thru 0x03737b (0x49a bytes) 000714 1789636420.511985 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38fc6): Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38fc6): Mirror 12: 0xf1042ed1 (stale) Mirror 13: 0xf1042ed1 (stale) CRC-32 checksum value for chunk [0, 0x38fc6): Mirror 1: 0xd498fdb8 Mirror 14: 0xd498fdb8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38fc6: skipped 000715 1789636421.206859 mapread 0x02e684 thru 0x038fc5 (0xa942 bytes) 000716 1789636421.235566 mapwrite 0x0220fb thru 0x02e72b (0xc631 bytes) 000717 1789636421.395217 mapwrite 0x037455 thru 0x03897a (0x1526 bytes) 000718 1789636421.429571 fallocate 0x002485 thru 0x007d76 (0x58f2 bytes) 000720 1789636421.436720 mapwrite 0x02bbee thru 0x03199d (0x5db0 bytes) 000721 1789636421.465362 mapread 0x033acd thru 0x038fc5 (0x54f9 bytes) 000722 1789636421.481177 MIRROR_RESYNC 4 mirrors 000723 1789636422.527538 write 0x015648 thru 0x02400b (0xe9c4 bytes) 000724 1789636422.585359 mapwrite 0x03b570 thru 0x03f2b8 (0x3d49 bytes) 000725 1789636422.688081 read 0x0009db thru 0x0097ba (0x8de0 bytes) 000726 1789636422.711581 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f2b9): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2b9): Mirror 12: 0x3c191848 (stale) Mirror 13: 0x3c191848 (stale) Mirror 14: 0x3c191848 (stale) CRC-32 checksum value for chunk [0, 0x3f2b9): Mirror 1: 0xd52e64bf Mirror 15: 0xd52e64bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2b9: skipped 000728 1789636423.513272 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f2b9): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2b9): Mirror 12: 0x3c191848 (stale) Mirror 13: 0x3c191848 (stale) Mirror 14: 0x3c191848 (stale) CRC-32 checksum value for chunk [0, 0x3f2b9): Mirror 1: 0xd52e64bf Mirror 15: 0xd52e64bf Mirror 16: 0xd52e64bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2b9: skipped 000729 1789636424.492606 write 0x03aaad thru 0x03ffff (0x5553 bytes) 000731 1789636424.537592 trunc from 0x040000 to 0x022c04 (0x1d3fb bytes) 000732 1789636424.561770 fallocate 0x002ab9 thru 0x012510 (0xfa58 bytes) 000733 1789636424.568172 write 0x00b6f5 thru 0x00c81f (0x112b bytes) 000734 1789636424.579220 mapwrite 0x005d62 thru 0x00a502 (0x47a1 bytes) 000735 1789636424.626842 trunc from 0x017810 to 0x01f0d8 (0x78c9 bytes) 000736 1789636424.632094 mapread 0x00aef6 thru 0x01773c (0xc847 bytes) 000737 1789636424.640541 read 0x018c26 thru 0x0199da (0xdb5 bytes) 000738 1789636424.641648 fallocate 0x024b00 thru 0x02a576 (0x5a77 bytes) 000739 1789636424.647779 write 0x005682 thru 0x006c9c (0x161b bytes) 000740 1789636424.650091 read 0x017b5b thru 0x021c7a (0xa120 bytes) 000741 1789636424.665402 MIRROR_RESYNC 6 mirrors 000742 1789636426.239118 MIRROR_RESYNC 6 mirrors 000743 1789636426.828092 write 0x035771 thru 0x03ffff (0xa88f bytes) 000744 1789636426.849878 trunc from 0x040000 to 0x01a0f0 (0x25f0f bytes) 000746 1789636426.862516 trunc from 0x01a0f1 to 0x02d2ea (0x131fa bytes) 000747 1789636426.872800 trunc from 0x013bd8 to 0x022397 (0xe7c0 bytes) 000748 1789636426.876890 mapread 0x00d232 thru 0x014b3c (0x790b bytes) 000749 1789636426.895925 MIRROR_RESYNC 6 mirrors 000750 1789636427.708409 fallocate 0x016f6c thru 0x01f2b2 (0x8347 bytes) 000751 1789636427.748239 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d2eb): Mirror 12: 0xe8ad63d7 (stale) Mirror 13: 0xe8ad63d7 (stale) Mirror 14: 0xe8ad63d7 (stale) Mirror 15: 0xe8ad63d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d2eb): Mirror 12: 0xe8ad63d7 (stale) Mirror 13: 0xe8ad63d7 (stale) Mirror 14: 0xe8ad63d7 (stale) Mirror 15: 0xe8ad63d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d2eb: skipped 000752 1789636428.062684 mapwrite 0x03711e thru 0x0374b0 (0x393 bytes) 000753 1789636428.092237 trunc from 0x01c1d8 to 0x027730 (0xb559 bytes) 000754 1789636428.099026 punch from 0x0270ba to 0x0279e0 (0x927 bytes) 000755 1789636428.106798 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x374b1): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x374b1): Mirror 12: 0x28f7aad0 (stale) Mirror 13: 0x28f7aad0 (stale) Mirror 14: 0x28f7aad0 (stale) Mirror 15: 0x28f7aad0 (stale) CRC-32 checksum value for chunk [0, 0x374b1): Mirror 1: 0xc9835761 Mirror 16: 0xc9835761 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x374b1: skipped 000756 1789636428.958013 read 0x02c122 thru 0x02cda8 (0xc87 bytes) 000757 1789636428.974572 trunc from 0x0169b2 to 0x017f6d (0x15bc bytes) 000758 1789636429.003887 trunc from 0x0374b1 to 0x000778 (0x36d38 bytes) 000759 1789636429.043885 read 0x00047f thru 0x000778 (0x2fa bytes) 000761 1789636429.055808 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x779): Mirror 12: 0x542a42cf (stale) Mirror 13: 0x542a42cf (stale) Mirror 14: 0x542a42cf (stale) Mirror 15: 0x542a42cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x779): Mirror 12: 0x542a42cf (stale) Mirror 13: 0x542a42cf (stale) Mirror 14: 0x542a42cf (stale) Mirror 15: 0x542a42cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x779: skipped 000762 1789636429.460265 trunc from 0x0004a6 to 0x000778 (0x2d3 bytes) 000763 1789636429.465785 fallocate 0x037dac thru 0x03abc5 (0x2e1a bytes) 000764 1789636429.477789 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x779): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x779): Mirror 12: 0x542a42cf (stale) Mirror 13: 0x542a42cf (stale) Mirror 14: 0x542a42cf (stale) Mirror 15: 0x542a42cf (stale) CRC-32 checksum value for chunk [0, 0x779): Mirror 1: 0x542a42cf Mirror 16: 0x542a42cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x779: skipped 000765 1789636430.133575 punch from 0x0003a3 to 0x000778 (0x3d6 bytes) 000767 1789636430.171846 trunc from 0x000779 to 0x0043c9 (0x3c51 bytes) 000768 1789636430.216379 trunc from 0x00040a to 0x0043c9 (0x3fc0 bytes) 000769 1789636430.222879 write 0x02897c thru 0x031229 (0x88ae bytes) 000770 1789636430.229859 read 0x01203e thru 0x01f777 (0xd73a bytes) 000771 1789636430.245222 read 0x0094bf thru 0x014c0a (0xb74c bytes) 000772 1789636430.270609 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3122a): Mirror 13: 0x770fde7e (stale) Mirror 14: 0x770fde7e (stale) Mirror 15: 0x770fde7e (stale) Mirror 16: 0xce645103 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3122a): Mirror 13: 0x770fde7e (stale) Mirror 14: 0x770fde7e (stale) Mirror 15: 0x770fde7e (stale) Mirror 16: 0xce645103 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3122a: skipped 000773 1789636430.668441 mapwrite 0x00d784 thru 0x012537 (0x4db4 bytes) 000774 1789636430.737829 mapwrite 0x028bd6 thru 0x02ab3f (0x1f6a bytes) 000776 1789636430.771484 mapread 0x02ae17 thru 0x031229 (0x6413 bytes) 000777 1789636430.779579 mapwrite 0x0360b1 thru 0x039625 (0x3575 bytes) 000778 1789636430.899360 MIRROR_RESYNC 5 mirrors 000779 1789636432.361665 trunc from 0x039626 to 0x03a377 (0xd52 bytes) 000780 1789636432.433815 read 0x00c390 thru 0x01426e (0x7edf bytes) 000781 1789636432.448702 trunc from 0x03a378 to 0x003e59 (0x3651e bytes) 000782 1789636432.479441 mapwrite 0x020e44 thru 0x021253 (0x410 bytes) 000783 1789636432.526394 fallocate 0x01f7d7 thru 0x026acf (0x72f9 bytes) 000784 1789636432.535726 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x26ad0): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26ad0): Mirror 13: 0xfcb0e595 (stale) Mirror 14: 0xfcb0e595 (stale) Mirror 15: 0xfcb0e595 (stale) Mirror 16: 0xfcb0e595 (stale) CRC-32 checksum value for chunk [0, 0x26ad0): Mirror 1: 0xfb665ba8 Mirror 17: 0xfb665ba8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26ad0: skipped 000785 1789636433.497564 mapread 0x017fd5 thru 0x0227c7 (0xa7f3 bytes) 000786 1789636433.549279 read 0x016a34 thru 0x01a206 (0x37d3 bytes) 000787 1789636433.558645 mapwrite 0x00a130 thru 0x00c6c4 (0x2595 bytes) 000788 1789636433.662028 trunc from 0x026ad0 to 0x01a8cc (0xc203 bytes) 000789 1789636433.712088 trunc from 0x01a8cd to 0x008525 (0x123a7 bytes) 000790 1789636433.735985 fallocate 0x0275e3 thru 0x035732 (0xe150 bytes) 000791 1789636433.741329 fallocate 0x00e9d6 thru 0x01a24e (0xb879 bytes) 000792 1789636433.747187 write 0x00c440 thru 0x0140ed (0x7cae bytes) 000793 1789636433.759246 fallocate 0x0318c1 thru 0x031bef (0x32f bytes) 000794 1789636433.767423 punch from 0x009590 to 0x012c40 (0x96b1 bytes) 000795 1789636433.793398 fallocate 0x026e6e thru 0x02d2e4 (0x6477 bytes) 000797 1789636433.806801 read 0x0148f8 thru 0x01f48d (0xab96 bytes) 000799 1789636433.830181 read 0x0106a6 thru 0x01c736 (0xc091 bytes) 000800 1789636433.830914 fallocate 0x014f62 thru 0x023d11 (0xedb0 bytes) 000801 1789636433.838475 read 0x0094da thru 0x009cef (0x816 bytes) 000802 1789636433.839021 trunc from 0x02d2e5 to 0x0209c7 (0xc91d bytes) 000804 1789636433.855679 trunc from 0x0209c8 to 0x00ef94 (0x11a33 bytes) 000805 1789636433.873746 trunc from 0x00ef95 to 0x00e089 (0xf0b bytes) 000806 1789636433.892225 trunc from 0x006fb2 to 0x00e089 (0x70d8 bytes) 000807 1789636433.896891 fallocate 0x019eb0 thru 0x026d48 (0xce99 bytes) 000808 1789636433.901132 mapwrite 0x01e4bd thru 0x026738 (0x827c bytes) 000809 1789636433.935675 mapread 0x021459 thru 0x026738 (0x52e0 bytes) 000810 1789636433.936881 trunc from 0x008e57 to 0x015c14 (0xcdbe bytes) 000811 1789636433.943746 trunc from 0x01165e to 0x01b3a1 (0x9d44 bytes) 000812 1789636433.950227 read 0x017114 thru 0x0195d8 (0x24c5 bytes) 000813 1789636433.972517 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x26739): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26739): Mirror 13: 0x150861e3 (stale) Mirror 14: 0x150861e3 (stale) Mirror 15: 0x150861e3 (stale) Mirror 16: 0x150861e3 (stale) Mirror 17: 0x2131be83 (stale) CRC-32 checksum value for chunk [0, 0x26739): Mirror 1: 0x6bdbb9db Mirror 18: 0x6bdbb9db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26739: skipped 000814 1789636434.850674 fallocate 0x01b760 thru 0x024239 (0x8ada bytes) 000815 1789636434.901336 trunc from 0x026739 to 0x006c05 (0x1fb33 bytes) 000816 1789636434.940542 read 0x0004da thru 0x006c05 (0x672c bytes) 000817 1789636434.966134 trunc from 0x004f7c to 0x006c05 (0x1c8a bytes) 000818 1789636434.979389 read 0x002180 thru 0x006c05 (0x4a86 bytes) 000819 1789636434.987992 trunc from 0x000329 to 0x006c05 (0x68dd bytes) 000820 1789636434.997204 trunc from 0x0016f0 to 0x006c05 (0x5516 bytes) 000821 1789636435.001480 trunc from 0x00426c to 0x006c05 (0x299a bytes) 000822 1789636435.015583 write 0x016723 thru 0x0182aa (0x1b88 bytes) 000823 1789636435.016785 write 0x02035c thru 0x02edb0 (0xea55 bytes) 000824 1789636435.019314 write 0x0094ff thru 0x010b96 (0x7698 bytes) 000825 1789636435.025754 trunc from 0x0138bd to 0x016741 (0x2e85 bytes) 000826 1789636435.045490 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2edb1): Mirror 14: 0x145e5f0a (stale) Mirror 15: 0x145e5f0a (stale) Mirror 16: 0x145e5f0a (stale) Mirror 17: 0x35a110b (stale) Mirror 18: 0x1bea45c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edb1): Mirror 14: 0x145e5f0a (stale) Mirror 15: 0x145e5f0a (stale) Mirror 16: 0x145e5f0a (stale) Mirror 17: 0x35a110b (stale) Mirror 18: 0x1bea45c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edb1: skipped 000828 1789636435.385483 trunc from 0x010217 to 0x019b64 (0x994e bytes) 000829 1789636435.389510 mapwrite 0x0084ba thru 0x0097bb (0x1302 bytes) 000830 1789636435.405795 mapwrite 0x025e41 thru 0x027c8a (0x1e4a bytes) 000831 1789636435.420589 write 0x01db55 thru 0x023515 (0x59c1 bytes) 000832 1789636435.429710 fallocate 0x024baf thru 0x026c83 (0x20d5 bytes) 000833 1789636435.442991 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2edb1): Mirror 15: 0x145e5f0a (stale) Mirror 16: 0x145e5f0a (stale) Mirror 17: 0x35a110b (stale) Mirror 18: 0x1bea45c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edb1): Mirror 15: 0x145e5f0a (stale) Mirror 16: 0x145e5f0a (stale) Mirror 17: 0x35a110b (stale) Mirror 18: 0x1bea45c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edb1: skipped 000834 1789636435.788665 mapwrite 0x024d62 thru 0x0314cb (0xc76a bytes) 000835 1789636435.846470 trunc from 0x018269 to 0x018914 (0x6ac bytes) 000837 1789636435.855746 read 0x005979 thru 0x00619d (0x825 bytes) 000838 1789636435.864245 write 0x0119ba thru 0x01a720 (0x8d67 bytes) 000839 1789636435.878202 mapread 0x001ab1 thru 0x004468 (0x29b8 bytes) 000840 1789636435.897654 trunc from 0x016ec2 to 0x01e9bc (0x7afb bytes) 000841 1789636435.921762 mapread 0x02a53b thru 0x02d3c5 (0x2e8b bytes) 000842 1789636435.924075 fallocate 0x0228d8 thru 0x02fe41 (0xd56a bytes) 000843 1789636435.935565 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x314cc): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x314cc): Mirror 15: 0xcbd64d0f (stale) Mirror 16: 0xcbd64d0f (stale) Mirror 17: 0xfdcb9ed7 (stale) Mirror 18: 0xdbd1f509 (stale) CRC-32 checksum value for chunk [0, 0x314cc): Mirror 1: 0xa31a7bf7 Mirror 19: 0xa31a7bf7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x314cc: skipped 000844 1789636436.891513 read 0x01baa8 thru 0x022206 (0x675f bytes) 000845 1789636436.919449 punch from 0x02213d to 0x02d808 (0xb6cc bytes) 000846 1789636436.991482 mapwrite 0x033857 thru 0x03ffff (0xc7a9 bytes) 000847 1789636437.072470 trunc from 0x02f006 to 0x033748 (0x4743 bytes) 000848 1789636437.078999 fallocate 0x002f84 thru 0x009ade (0x6b5b bytes) 000849 1789636437.084121 trunc from 0x020572 to 0x02388f (0x331e bytes) 000850 1789636437.095281 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x36bd86da (stale) Mirror 16: 0x36bd86da (stale) Mirror 17: 0xc1dd65b7 (stale) Mirror 18: 0x14fd131e (stale) Mirror 19: 0x8911930f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xae9a5fa5 Mirror 20: 0xae9a5fa5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000851 1789636438.076155 trunc from 0x0072a2 to 0x014f2e (0xdc8d bytes) 000852 1789636438.131195 read 0x03f84c thru 0x03ffff (0x7b4 bytes) 000853 1789636438.135763 punch from 0x023fbf to 0x033bf8 (0xfc3a bytes) 000854 1789636438.144890 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x36bd86da (stale) Mirror 16: 0x36bd86da (stale) Mirror 17: 0xc1dd65b7 (stale) Mirror 18: 0x14fd131e (stale) Mirror 19: 0x8911930f (stale) Mirror 20: 0xae9a5fa5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x907ca18a Mirror 21: 0x907ca18a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000855 1789636439.048152 MIRROR_RESYNC 8 mirrors 000856 1789636440.728943 trunc from 0x040000 to 0x0150d4 (0x2af2b bytes) 000857 1789636440.776131 mapwrite 0x011290 thru 0x0131db (0x1f4c bytes) 000859 1789636440.853498 punch from 0x007676 to 0x00edab (0x7736 bytes) 000860 1789636440.868538 trunc from 0x0150d5 to 0x038d18 (0x23c44 bytes) 000861 1789636440.906428 fallocate 0x0028e3 thru 0x003ded (0x150b bytes) 000862 1789636440.913495 trunc from 0x038d19 to 0x03b0e6 (0x23ce bytes) 000863 1789636440.932125 fallocate 0x01f00a thru 0x02a19a (0xb191 bytes) 000864 1789636440.944379 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b0e7): Mirror 16: 0x6c078304 (stale) Mirror 17: 0x6c078304 (stale) Mirror 18: 0x6c078304 (stale) Mirror 19: 0x6c078304 (stale) Mirror 20: 0x6c078304 (stale) Mirror 21: 0x6c078304 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b0e7): Mirror 16: 0x6c078304 (stale) Mirror 17: 0x6c078304 (stale) Mirror 18: 0x6c078304 (stale) Mirror 19: 0x6c078304 (stale) Mirror 20: 0x6c078304 (stale) Mirror 21: 0x6c078304 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b0e7: skipped 000865 1789636441.424213 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b0e7): Mirror 17: 0x6c078304 (stale) Mirror 18: 0x6c078304 (stale) Mirror 19: 0x6c078304 (stale) Mirror 20: 0x6c078304 (stale) Mirror 21: 0x6c078304 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b0e7): Mirror 17: 0x6c078304 (stale) Mirror 18: 0x6c078304 (stale) Mirror 19: 0x6c078304 (stale) Mirror 20: 0x6c078304 (stale) Mirror 21: 0x6c078304 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b0e7: skipped 000866 1789636441.777395 MIRROR_RESYNC 6 mirrors 000867 1789636443.108447 trunc from 0x0353ba to 0x03b0e6 (0x5d2d bytes) 000868 1789636443.170642 read 0x0029e4 thru 0x00e1f4 (0xb811 bytes) 000869 1789636443.186510 trunc from 0x03b0e7 to 0x0141ea (0x26efc bytes) 000871 1789636443.219560 fallocate 0x00a566 thru 0x016914 (0xc3af bytes) 000872 1789636443.227394 mapwrite 0x028216 thru 0x02a34d (0x2138 bytes) 000873 1789636443.288001 mapread 0x0030b2 thru 0x00bf37 (0x8e86 bytes) 000874 1789636443.289499 mapread 0x026a87 thru 0x02a34d (0x38c7 bytes) 000875 1789636443.294010 MIRROR_RESYNC 6 mirrors 000876 1789636444.650458 trunc from 0x023210 to 0x025599 (0x238a bytes) 000878 1789636444.673135 read 0x0127d3 thru 0x0129ee (0x21c bytes) 000879 1789636444.676486 mapread 0x008fc3 thru 0x016e93 (0xded1 bytes) 000880 1789636444.691642 read 0x01df9b thru 0x01e6a1 (0x707 bytes) 000881 1789636444.697947 fallocate 0x031dbf thru 0x03a76a (0x89ac bytes) 000882 1789636444.704169 write 0x004390 thru 0x00581e (0x148f bytes) 000883 1789636444.705590 punch from 0x026978 to 0x02a34d (0x39d6 bytes) 000884 1789636444.711574 trunc from 0x011921 to 0x012bc2 (0x12a2 bytes) 000885 1789636444.720412 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2a34e): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a34e): Mirror 17: 0x2e4aa1fa (stale) Mirror 18: 0x2e4aa1fa (stale) Mirror 19: 0x2e4aa1fa (stale) Mirror 20: 0x2e4aa1fa (stale) Mirror 21: 0x2e4aa1fa (stale) CRC-32 checksum value for chunk [0, 0x2a34e): Mirror 1: 0x44a60d7e Mirror 22: 0x44a60d7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a34e: skipped 000886 1789636445.665961 fallocate 0x0015e5 thru 0x009700 (0x811c bytes) 000887 1789636445.700286 read 0x01c5bd thru 0x025a69 (0x94ad bytes) 000888 1789636445.708670 mapread 0x0179b0 thru 0x01ce1b (0x546c bytes) 000889 1789636445.727629 fallocate 0x0004a2 thru 0x00c89b (0xc3fa bytes) 000890 1789636445.732894 punch from 0x00e86e to 0x013b1b (0x52ae bytes) 000891 1789636445.738023 write 0x02c98c thru 0x038d48 (0xc3bd bytes) 000892 1789636445.743344 mapread 0x03858a thru 0x038d48 (0x7bf bytes) 000893 1789636445.744625 read 0x01bcc5 thru 0x02220c (0x6548 bytes) 000894 1789636445.745560 trunc from 0x038d49 to 0x02be1c (0xcf2c bytes) 000895 1789636445.763334 mapread 0x029dd7 thru 0x02be1c (0x2046 bytes) 000896 1789636445.770197 trunc from 0x02be1d to 0x039c92 (0xde76 bytes) 000897 1789636445.786295 mapread 0x036d73 thru 0x039c92 (0x2f20 bytes) 000898 1789636445.794683 trunc from 0x022b2f to 0x0291dd (0x66af bytes) 000899 1789636445.799617 mapwrite 0x02419c thru 0x028b65 (0x49ca bytes) 000900 1789636445.842412 mapwrite 0x01b4e1 thru 0x023117 (0x7c37 bytes) 000901 1789636445.872589 punch from 0x015cfa to 0x022dd5 (0xd0dc bytes) 000902 1789636445.879087 read 0x010d23 thru 0x0153ff (0x46dd bytes) 000903 1789636445.890650 mapwrite 0x0132ee thru 0x0137b7 (0x4ca bytes) 000904 1789636445.901154 mapread 0x011999 thru 0x01627c (0x48e4 bytes) 000906 1789636445.903097 write 0x02f87a thru 0x03318e (0x3915 bytes) 000907 1789636445.904409 write 0x01e896 thru 0x027eb7 (0x9622 bytes) 000908 1789636445.905926 trunc from 0x039c93 to 0x03b48b (0x17f9 bytes) 000910 1789636445.924838 fallocate 0x0146ea thru 0x016498 (0x1daf bytes) 000911 1789636445.931991 trunc from 0x03b48c to 0x0083cf (0x330bc bytes) 000912 1789636445.953950 write 0x01c9db thru 0x01e5a3 (0x1bc9 bytes) 000913 1789636445.955924 read 0x00cce4 thru 0x01b891 (0xebae bytes) 000914 1789636445.963307 read 0x016fec thru 0x01e5a3 (0x75b8 bytes) 000915 1789636445.964058 mapwrite 0x0071fe thru 0x009e3e (0x2c41 bytes) 000916 1789636445.982476 trunc from 0x00ba0e to 0x014e27 (0x941a bytes) 000917 1789636445.987907 trunc from 0x01e5a4 to 0x03eba5 (0x20602 bytes) 000918 1789636446.002996 write 0x01f300 thru 0x0224e4 (0x31e5 bytes) 000919 1789636446.025595 mapread 0x022c00 thru 0x0247a7 (0x1ba8 bytes) 000920 1789636446.026849 fallocate 0x02c51b thru 0x034f05 (0x89eb bytes) 000922 1789636446.031679 write 0x000d17 thru 0x003cff (0x2fe9 bytes) 000925 1789636446.034727 trunc from 0x019e93 to 0x01b024 (0x1192 bytes) 000926 1789636446.039188 mapwrite 0x033e86 thru 0x03f4d2 (0xb64d bytes) 000927 1789636446.082352 MIRROR_RESYNC 7 mirrors 000928 1789636447.617531 mapwrite 0x01456e thru 0x01a232 (0x5cc5 bytes) 000929 1789636447.698400 trunc from 0x01eaa1 to 0x02c1a8 (0xd708 bytes) 000930 1789636447.708415 MIRROR_RESYNC 7 mirrors 000931 1789636449.355168 mapwrite 0x00366f thru 0x0039d2 (0x364 bytes) 000932 1789636449.422981 punch from 0x01c15f to 0x025e15 (0x9cb7 bytes) 000933 1789636449.428085 punch from 0x021661 to 0x0247a7 (0x3147 bytes) 000934 1789636449.435779 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f4d3): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f4d3): Mirror 17: 0xf0dbaed (stale) Mirror 18: 0xf0dbaed (stale) Mirror 19: 0xf0dbaed (stale) Mirror 20: 0xf0dbaed (stale) Mirror 21: 0xf0dbaed (stale) Mirror 22: 0xf0dbaed (stale) CRC-32 checksum value for chunk [0, 0x3f4d3): Mirror 1: 0x8270005a Mirror 23: 0x8270005a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f4d3: skipped 000935 1789636450.132441 punch from 0x03e2bd to 0x03f4d2 (0x1216 bytes) 000936 1789636450.158705 fallocate 0x039466 thru 0x03d023 (0x3bbe bytes) 000937 1789636450.163355 write 0x03e524 thru 0x03ffff (0x1adc bytes) 000938 1789636450.171721 punch from 0x02c179 to 0x037f91 (0xbe19 bytes) 000939 1789636450.179810 trunc from 0x040000 to 0x01f97b (0x20684 bytes) 000940 1789636450.206790 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f97c): Mirror 17: 0xf8d83e11 (stale) Mirror 18: 0xf8d83e11 (stale) Mirror 19: 0xf8d83e11 (stale) Mirror 20: 0xf8d83e11 (stale) Mirror 21: 0xf8d83e11 (stale) Mirror 23: 0x432d47f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f97c): Mirror 17: 0xf8d83e11 (stale) Mirror 18: 0xf8d83e11 (stale) Mirror 19: 0xf8d83e11 (stale) Mirror 20: 0xf8d83e11 (stale) Mirror 21: 0xf8d83e11 (stale) Mirror 23: 0x432d47f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f97c: skipped 000942 1789636450.592459 read 0x01b80a thru 0x01f97b (0x4172 bytes) 000943 1789636450.601604 trunc from 0x0081a5 to 0x010d13 (0x8b6f bytes) 000944 1789636450.605785 trunc from 0x01f97c to 0x0044a0 (0x1b4db bytes) 000945 1789636450.624969 trunc from 0x0044a1 to 0x0356c6 (0x31226 bytes) 000946 1789636450.643364 write 0x036464 thru 0x038d85 (0x2922 bytes) 000947 1789636450.644824 trunc from 0x026a66 to 0x035c5b (0xf1f6 bytes) 000948 1789636450.650317 read 0x013679 thru 0x017fd5 (0x495d bytes) 000949 1789636450.662831 trunc from 0x038d86 to 0x022e29 (0x15f5c bytes) 000950 1789636450.680122 fallocate 0x02fe4b thru 0x03d09a (0xd250 bytes) 000952 1789636450.689981 mapread 0x0179ba thru 0x01fe08 (0x844f bytes) 000953 1789636450.697251 trunc from 0x022e2a to 0x0115ba (0x1186f bytes) 000954 1789636450.715890 punch from 0x00fd0d to 0x0115ba (0x18ae bytes) 000955 1789636450.720956 mapread 0x00070b thru 0x0099b9 (0x92af bytes) 000956 1789636450.723779 mapread 0x006623 thru 0x00e1da (0x7bb8 bytes) 000958 1789636450.729898 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x115bb): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x115bb): Mirror 17: 0x423c41b7 (stale) Mirror 18: 0x423c41b7 (stale) Mirror 19: 0x423c41b7 (stale) Mirror 20: 0x423c41b7 (stale) Mirror 21: 0x423c41b7 (stale) Mirror 23: 0xa446d6c2 (stale) CRC-32 checksum value for chunk [0, 0x115bb): Mirror 1: 0x9df457b Mirror 24: 0x9df457b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x115bb: skipped 000959 1789636451.659101 trunc from 0x00dd55 to 0x0115ba (0x3866 bytes) 000960 1789636451.705852 punch from 0x005676 to 0x00f0e8 (0x9a73 bytes) 000961 1789636451.722402 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x115bb): Mirror 17: 0x423c41b7 (stale) Mirror 18: 0x423c41b7 (stale) Mirror 19: 0x423c41b7 (stale) Mirror 21: 0x423c41b7 (stale) Mirror 23: 0xa446d6c2 (stale) Mirror 24: 0x9df457b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x115bb): Mirror 17: 0x423c41b7 (stale) Mirror 18: 0x423c41b7 (stale) Mirror 19: 0x423c41b7 (stale) Mirror 21: 0x423c41b7 (stale) Mirror 23: 0xa446d6c2 (stale) Mirror 24: 0x9df457b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x115bb: skipped 000962 1789636452.097376 read 0x005b8e thru 0x0115ba (0xba2d bytes) 000963 1789636452.108190 mapread 0x00414b thru 0x00bd27 (0x7bdd bytes) 000964 1789636452.117531 write 0x038fc0 thru 0x03ffff (0x7040 bytes) 000966 1789636452.127335 trunc from 0x01aa6b to 0x020292 (0x5828 bytes) 000967 1789636452.134779 fallocate 0x0152db thru 0x022382 (0xd0a8 bytes) 000968 1789636452.140265 trunc from 0x040000 to 0x0200d8 (0x1ff27 bytes) 000969 1789636452.157994 trunc from 0x0200d9 to 0x03629a (0x161c2 bytes) 000970 1789636452.173323 fallocate 0x02191c thru 0x02a33c (0x8a21 bytes) 000971 1789636452.179349 trunc from 0x03629b to 0x02c902 (0x9998 bytes) 000972 1789636452.202147 punch from 0x024616 to 0x02c902 (0x82ed bytes) 000973 1789636452.208304 trunc from 0x010ae2 to 0x0129e9 (0x1f08 bytes) 000974 1789636452.215242 read 0x024d31 thru 0x025d05 (0xfd5 bytes) 000975 1789636452.235512 MIRROR_RESYNC 7 mirrors 000976 1789636453.357781 mapread 0x00c1fe thru 0x0174ed (0xb2f0 bytes) 000977 1789636453.388350 read 0x00214c thru 0x00c78c (0xa641 bytes) 000978 1789636453.395730 trunc from 0x02c903 to 0x036ff1 (0xa6ef bytes) 000979 1789636453.471603 MIRROR_RESYNC 7 mirrors 000980 1789636454.360553 mapread 0x01a070 thru 0x01db2d (0x3abe bytes) 000981 1789636454.391018 MIRROR_RESYNC 7 mirrors 000983 1789636454.943438 read 0x031f54 thru 0x036ff1 (0x509e bytes) 000985 1789636454.956000 mapread 0x01fa04 thru 0x023e83 (0x4480 bytes) 000986 1789636454.956932 mapread 0x036bcc thru 0x036ff1 (0x426 bytes) 000987 1789636454.958606 punch from 0x008d5c to 0x016e60 (0xe105 bytes) 000988 1789636455.003353 mapread 0x00d30b thru 0x00ff30 (0x2c26 bytes) 000990 1789636455.023565 fallocate 0x02a3da thru 0x037562 (0xd189 bytes) 000991 1789636455.028553 write 0x03dc94 thru 0x03ffff (0x236c bytes) 000992 1789636455.030890 fallocate 0x006119 thru 0x00b9a2 (0x588a bytes) 000994 1789636455.037232 read 0x01cddc thru 0x02ad2a (0xdf4f bytes) 000995 1789636455.050871 trunc from 0x01798b to 0x01d56f (0x5be5 bytes) 000996 1789636455.056867 fallocate 0x0082df thru 0x00eb59 (0x687b bytes) 000997 1789636455.061276 trunc from 0x010a82 to 0x016541 (0x5ac0 bytes) 000998 1789636455.066315 trunc from 0x0267af to 0x031c8e (0xb4e0 bytes) 000999 1789636455.070535 read 0x00ba21 thru 0x019c8c (0xe26c bytes) 001000 1789636455.078450 fallocate 0x02c5a0 thru 0x03479e (0x81ff bytes) All operations completed A-OK! PASS 70a (89s) == sanity-flr test 71: check mirror extend parent fid ==== 05:14:19 (1789636459) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x5d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 05:14:27 (1789636467) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224845 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3d8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748367 s, 14.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3d8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3d7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3d8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3d7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3d9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 05:14:34 (1789636474) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre2 Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7402464 .. Extending file size to 3777856 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9181184 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7919200 .. Extending file size to 8231936 .. Extending file size to 8459936 .. Extending file size to 8635328 .. Extending file size to 8645216 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9347136 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 61768 61770 61771 61772 61773 failed 192.168.202.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre2 (opts:) 192.168.202.115@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:15:51 (1789636551) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre2 Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1391680 .. Extending file size to 8216000 .. done Extending file size to 3520128 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7240928 .. Extending file size to 8568096 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8403072 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8810400 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8778720 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63432 63433 63434 63437 63438 done 192.168.202.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre2 (opts:) 192.168.202.115@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:17:10 (1789636630) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2572 1268732 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17464 3589556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24884 3582136 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42348 7171692 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2572 1268732 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17464 3589556 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24884 3582136 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42348 7171692 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0897473 s, 23.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170214 s, 12.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xb923dbf0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:17:28 (1789636648) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0753616 s, 27.8 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 05:17:34 (1789636654) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x279:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3ed:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3ed:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3ee:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3ee:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:17:40 (1789636660) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0549758 s, 38.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:17:46 (1789636666) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0837808 s, 25.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3ef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3ef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3f0:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:17:53 (1789636673) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104139 s, 20.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3f1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3f1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3f2:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:18:24 (1789636704) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg215-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054273 s, 19.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4f:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:19:10 (1789636750) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg215-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0731178 s, 28.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:19:55 (1789636795) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg215-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg215-server: Pool lustre.test_204f created Adding targets to pool oleg215-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg215-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109741 s, 19.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg215-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg215-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg215-server: Pool lustre.test_204f destroyed PASS 204f (63s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:20:58 (1789636858) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:21:05 (1789636865) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:21:12 (1789636872) oleg215-server: Pool lustre.test_206 created Adding targets to pool oleg215-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg215-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg215-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg215-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg215-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:21:36 (1789636896) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648094 s, 16.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435109 s, 24.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf46c2156 Mirror 3: 0xf46c2156 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:21:45 (1789636905) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:21:47 (1789636907) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:21:50 (1789636910) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0696834 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x810000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x25b0000 Read this (4 bytes): BBBB 192.168.202.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre2 (opts:) PASS 209a (31s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:22:21 (1789636941) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0361726 s, 226 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:22:27 (1789636947) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261893 s, 40.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x29a lmm_fid: [0x200000405:0x29a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x29a lmm_fid: [0x200000405:0x29a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3ff:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611802 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-17 05:22:28.000000000 -0400 Modify: 2026-09-17 05:22:28.000000000 -0400 Change: 2026-09-17 05:22:29.000000000 -0400 Birth: 2026-09-17 05:22:28.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:22:34 (1789636954) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383283 s, 27.4 MB/s oleg215-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg215-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg215-server: O/280000401/d30/3fe: File not found by ext2_lookup PASS 210b (19s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:22:53 (1789636973) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.138427 s, 296 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.020361 s, 201 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 17 05:22 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 17 05:22 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:23:00 (1789636980) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:23:06 (1789636986) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1682 sec =========== 05:23:13 (1789636993) === sanity-flr: start cleanup 05:23:14 (1789636994) === === sanity-flr: finish cleanup 05:23:21 (1789637001) ===