-----============= acceptance-small: sanity-flr ============----- Tue Jun 23 14:59:22 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg340-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 14:59:40 (1782241180) === oleg340-client.virtnet: executing check_config_client /mnt/lustre oleg340-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg340-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a89d00cf000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a89d00cf000.idle_timeout=debug disable quota as required oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:00:19 (1782241219) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8219] == sanity-flr test 0a: lfs mirror create with -N option == 15:00:22 (1782241222) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:00:41 (1782241241) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:00:45 (1782241245) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:00:48 (1782241248) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:01:12 (1782241272) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:01:16 (1782241276) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:01:19 (1782241279) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8a89d00cf000.stats=clear osc.lustre-OST0001-osc-ffff8a89d00cf000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:01:27 (1782241287) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:01:37 (1782241297) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:01:44 (1782241304) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:01:48 (1782241308) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:01:56 (1782241316) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 15:02:05 (1782241325) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:02:12 (1782241332) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:02:21 (1782241341) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:02:29 (1782241349) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:02:41 (1782241361) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:02:47 (1782241367) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.61625 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106315 s, 9.9 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:02:57 (1782241377) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.437351 s, 40.8 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:03:07 (1782241387) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (14s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:03:21 (1782241401) Stopping /mnt/lustre-ost1 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a89d00cf000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a89d00cf000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg340-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a89d00cf000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a89d00cf000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg340-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:04:32 (1782241472) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.236142 s, 13.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a89d00cf000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a89d00cf000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg340-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:05:15 (1782241515) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a89d00cf000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a89d00cf000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg340-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a89d00cf000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a89d00cf000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg340-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:06:37 (1782241597) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.99702 s, 17.5 MB/s -rw-r--r-- 1 root root 52428800 Jun 23 15:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.736s user 0m0.001s sys 0m0.196s Stopping /mnt/lustre-ost1 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a89d00cf000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a89d00cf000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.043s user 0m0.000s sys 0m0.170s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg340-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a89d00cf000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a89d00cf000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.116s user 0m0.002s sys 0m0.205s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg340-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg340-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a89d00cf000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a89d00cf000.ost_server_uuid in FULL state after 0 sec PASS 33b (84s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:08:01 (1782241681) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:08:05 (1782241685) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:08:08 (1782241688) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 15:08:11 (1782241691) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 15:08:19 (1782241699) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.724754 s, 17.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:08:32 (1782241712) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301913 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360533 s, 29.1 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:08:39 (1782241719) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:08:47 (1782241727) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00226558 s, 452 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00517879 s, 198 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00464839 s, 220 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00222885 s, 459 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 15:08:59 (1782241739) create test files with size 15746863 .. 129e480815b018643364f2d62ed608eb - 7f2b0a44159b3beac75ac68b91999105 - 714ecf3292fe53c19a12b450b00e17ee - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 15:09:22 (1782241762) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:09:37 (1782241777) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:09:44 (1782241784) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665452 s, 15.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370504 s, 28.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 15:09:53 (1782241793) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0875787 s, 23.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.271873 s, 15.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178419 s, 11.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 15:10:07 (1782241807) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 15:10:12 (1782241812) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:10:15 (1782241815) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365301 s, 28.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 15:10:24 (1782241824) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.36563 s, 8.6 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:10:34 (1782241834) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:10:44 (1782241844) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.304612 s, 34.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:10:54 (1782241854) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8a89c7db8000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.441603 s, 23.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8a89c89f3800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:11:10 (1782241870) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:11:47 (1782241907) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:12:24 (1782241944) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 15:12:28 (1782241948) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.191103 s, 16.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.140891 s, 22.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x64e10ced (nosync,stale) Mirror 2: 0x64e10ced (nosync,stale) ** verify mirror 2 doesn't change original checksum: 20189b1e98e32a0a255efa20dca8dc65 - mirror 1 checksum: f45d419e0e04d5d97a8f29715c284b5f - mirror 2 checksum: 20189b1e98e32a0a255efa20dca8dc65 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-23T15:12:32-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-23T15:12:32-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 20189b1e98e32a0a255efa20dca8dc65 - mirror 1 checksum: f45d419e0e04d5d97a8f29715c284b5f - mirror 2 checksum: f45d419e0e04d5d97a8f29715c284b5f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-23T15:12:32-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-23T15:12:32-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 15:12:38 (1782241958) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037159 s, 28.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:12:46 (1782241966) Failing mds1 on oleg340-server Stopping /mnt/lustre-mds1 (opts:) on oleg340-server 15:12:57 (1782241977) shut down facet: mds1 facet_host: oleg340-server facet_failover_host: oleg340-server Failover mds1 to oleg340-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 15:13:27 (1782242007) targets are mounted 15:13:27 (1782242007) facet_failover done oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272125 s, 38.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 50A (58s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:13:43 (1782242023) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0606028 s, 338 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0291131 s, 703 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:13:54 (1782242034) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0692707 s, 296 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00820799 s, 2.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:14:03 (1782242043) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00971287 s, 1.7 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:14:12 (1782242052) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0820736 s, 250 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:14:20 (1782242060) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0173037 s, 237 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0506751 s, 80.8 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:14:28 (1782242068) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:14:51 (1782242091) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:15:21 (1782242121) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:15:47 (1782242147) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:16:12 (1782242172) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 15:16:21 (1782242181) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57379 mirror split pid 57381 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7768: 57379 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57381 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 15:17:29 (1782242249) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 2251 truncating to largest ever: 0x24218 000001 1782242251.598340 trunc from 00000000 to 0x024217 (0x24218 bytes) 000002 1782242251.628487 trunc from 0x01b516 to 0x024217 (0x8d02 bytes) truncating to largest ever: 0x341b0 000003 1782242251.635259 trunc from 0x024218 to 0x0341af (0xff98 bytes) 000004 1782242251.666101 punch from 0x017987 to 0x023a9b (0xc115 bytes) 000007 1782242251.676410 fallocate 0x01d44e thru 0x02a883 (0xd436 bytes) 000008 1782242251.689811 mapread 0x031ba1 thru 0x0341af (0x260f bytes) 000009 1782242251.740718 trunc from 0x028e7c to 0x0341af (0xb334 bytes) 000010 1782242251.750706 read 0x031c91 thru 0x0341af (0x251f bytes) 000011 1782242251.782677 mapread 0x029448 thru 0x0341af (0xad68 bytes) 000012 1782242251.810445 mapread 0x00b28b thru 0x012917 (0x768d bytes) 000013 1782242251.837333 write 0x025a29 thru 0x0307e7 (0xadbf bytes) 000014 1782242251.844443 fallocate 0x00b3ea thru 0x00c60b (0x1222 bytes) 000015 1782242251.849417 read 0x02604c thru 0x02a29d (0x4252 bytes) fallocating to largest ever: 0x40000 000016 1782242251.853012 fallocate 0x03f6eb thru 0x03ffff (0x915 bytes) 000017 1782242251.875938 trunc from 0x040000 to 0x01b58f (0x24a70 bytes) 000018 1782242251.927194 read 0x007147 thru 0x00c411 (0x52cb bytes) 000019 1782242251.928487 mapread 0x00bb44 thru 0x0161bd (0xa67a bytes) 000020 1782242251.941804 mapread 0x0188d2 thru 0x01b58f (0x2cbe bytes) 000021 1782242251.942984 trunc from 0x01b590 to 0x01e999 (0x340a bytes) 000022 1782242251.975848 write 0x0012c1 thru 0x005ab6 (0x47f6 bytes) 000023 1782242251.978913 trunc from 0x01e99a to 0x001238 (0x1d761 bytes) 000025 1782242251.999659 fallocate 0x00d730 thru 0x013d4d (0x661e bytes) 000027 1782242252.007556 trunc from 0x013d4e to 0x0172d2 (0x3585 bytes) 000029 1782242252.033197 trunc from 0x0172d3 to 0x03cdee (0x25b1c bytes) 000030 1782242252.054678 write 0x03226e thru 0x03530a (0x309d bytes) 000031 1782242252.085234 write 0x00a892 thru 0x00ca71 (0x21e0 bytes) 000032 1782242252.121599 mapread 0x009ad0 thru 0x00e123 (0x4654 bytes) 000033 1782242252.152574 trunc from 0x03cdef to 0x03a958 (0x2496 bytes) 000034 1782242252.174956 mapread 0x0004e4 thru 0x00a78e (0xa2ab bytes) 000035 1782242252.226743 write 0x005c5f thru 0x011b28 (0xbeca bytes) 000037 1782242252.239902 mapread 0x01e8dd thru 0x01ff74 (0x1698 bytes) 000039 1782242252.247622 punch from 0x022760 to 0x02874e (0x5fef bytes) 000041 1782242252.256508 mapwrite 0x014f40 thru 0x0188c5 (0x3986 bytes) 000042 1782242252.345254 mapread 0x022ced thru 0x030669 (0xd97d bytes) 000043 1782242252.382863 mapwrite 0x020b6b thru 0x029853 (0x8ce9 bytes) 000044 1782242252.406385 fallocate 0x03ad7e thru 0x03b3d3 (0x656 bytes) 000045 1782242252.412659 read 0x016ed5 thru 0x01740e (0x53a bytes) 000049 1782242252.416806 trunc from 0x03b3d4 to 0x002b26 (0x388ad bytes) 000050 1782242252.439236 mapread 0x002149 thru 0x002b26 (0x9de bytes) 000051 1782242252.441401 write 0x033997 thru 0x03b34a (0x79b4 bytes) 000052 1782242252.445861 punch from 0x01f8f4 to 0x026884 (0x6f91 bytes) 000054 1782242252.454387 mapwrite 0x00fcdc thru 0x017dc4 (0x80e9 bytes) 000055 1782242252.490527 punch from 0x016172 to 0x023da1 (0xdc30 bytes) 000056 1782242252.497388 read 0x011574 thru 0x017089 (0x5b16 bytes) 000057 1782242252.515151 mapwrite 0x016537 thru 0x01b2d6 (0x4da0 bytes) 000058 1782242252.540680 write 0x03ec72 thru 0x03ffff (0x138e bytes) 000059 1782242252.543441 read 0x0048f5 thru 0x011308 (0xca14 bytes) 000060 1782242252.545205 trunc from 0x040000 to 0x03d1eb (0x2e14 bytes) 000063 1782242252.570233 write 0x0062d7 thru 0x00d120 (0x6e4a bytes) 000064 1782242252.573230 punch from 0x0327fc to 0x03d1eb (0xa9f0 bytes) 000066 1782242252.580286 fallocate 0x026bde thru 0x0305a4 (0x99c7 bytes) 000068 1782242252.589143 trunc from 0x0371f0 to 0x03d1eb (0x5ffc bytes) 000069 1782242252.594913 mapwrite 0x010511 thru 0x01c567 (0xc057 bytes) 000070 1782242252.620329 mapread 0x031731 thru 0x03b8fa (0xa1ca bytes) 000071 1782242252.637411 mapwrite 0x006bec thru 0x011f65 (0xb37a bytes) 000072 1782242252.667555 fallocate 0x036077 thru 0x03aeb9 (0x4e43 bytes) 000073 1782242252.673766 trunc from 0x0160a3 to 0x0210b1 (0xb00f bytes) 000074 1782242252.679251 trunc from 0x0172a3 to 0x01e32c (0x708a bytes) 000075 1782242252.685139 fallocate 0x019b36 thru 0x022fc9 (0x9494 bytes) 000076 1782242252.690878 mapwrite 0x01f7c9 thru 0x024736 (0x4f6e bytes) 000077 1782242252.736243 trunc from 0x03d1ec to 0x02cc10 (0x105db bytes) 000081 1782242252.761182 read 0x015f01 thru 0x01c32c (0x642c bytes) 000082 1782242252.774121 read 0x00f68e thru 0x01da9b (0xe40e bytes) 000083 1782242252.776241 punch from 0x000f3c to 0x00f638 (0xe6fd bytes) 000084 1782242252.784290 read 0x01d3b5 thru 0x02cc10 (0xf85c bytes) 000086 1782242252.788271 read 0x003e6a thru 0x00c4bf (0x8656 bytes) 000087 1782242252.800878 write 0x02f99f thru 0x03eb80 (0xf1e2 bytes) 000088 1782242252.806884 trunc from 0x03eb81 to 0x03c7fe (0x2382 bytes) 000090 1782242252.827473 mapwrite 0x031573 thru 0x0336ca (0x2158 bytes) 000091 1782242252.850456 write 0x0399ad thru 0x03ffff (0x6653 bytes) 000092 1782242252.852609 read 0x01c08a thru 0x0226e4 (0x665b bytes) 000093 1782242252.853570 mapread 0x03874f thru 0x03ffff (0x78b1 bytes) 000094 1782242252.855409 mapread 0x01a7e3 thru 0x026fad (0xc7cb bytes) 000095 1782242252.857391 trunc from 0x03ba13 to 0x03ffff (0x45ed bytes) 000096 1782242252.873913 mapread 0x00cd56 thru 0x00ce2f (0xda bytes) 000097 1782242252.875337 trunc from 0x03dac4 to 0x03ffff (0x253c bytes) 000098 1782242252.880411 read 0x03c533 thru 0x03ffff (0x3acd bytes) 000099 1782242252.890410 trunc from 0x040000 to 0x0335b9 (0xca46 bytes) 000100 1782242252.906197 punch from 0x026b45 to 0x0292f9 (0x27b5 bytes) 000101 1782242252.918546 mapread 0x0290ac thru 0x0335b9 (0xa50e bytes) 000102 1782242252.930267 read 0x020bc4 thru 0x02c97f (0xbdbc bytes) 000103 1782242252.937604 write 0x0167a3 thru 0x023d38 (0xd596 bytes) 000104 1782242252.940528 fallocate 0x003e77 thru 0x012d12 (0xee9c bytes) 000105 1782242252.946404 trunc from 0x019edd to 0x01f0d8 (0x51fc bytes) 000106 1782242252.962546 mapread 0x01ddc9 thru 0x02c9d5 (0xec0d bytes) 000107 1782242252.978407 write 0x01c234 thru 0x029c70 (0xda3d bytes) 000108 1782242252.985881 read 0x02870e thru 0x02e1df (0x5ad2 bytes) 000109 1782242252.987322 read 0x02b2f6 thru 0x02e820 (0x352b bytes) 000110 1782242252.988474 mapwrite 0x034704 thru 0x03ec5e (0xa55b bytes) 000111 1782242253.041379 trunc from 0x03ec5f to 0x03352a (0xb734 bytes) 000112 1782242253.059448 mapread 0x0308ea thru 0x03352a (0x2c41 bytes) 000113 1782242253.061430 trunc from 0x03352b to 0x015341 (0x1e1e9 bytes) 000114 1782242253.084833 punch from 0x00d0a4 to 0x0135d1 (0x652e bytes) 000115 1782242253.093322 punch from 0x009ca8 to 0x00c8fe (0x2c57 bytes) 000116 1782242253.099575 write 0x022a63 thru 0x02c87e (0x9e1c bytes) 000117 1782242253.103392 read 0x004a38 thru 0x008b01 (0x40ca bytes) 000118 1782242253.105381 trunc from 0x00cb31 to 0x015a9a (0x8f6a bytes) 000119 1782242253.112875 trunc from 0x02c87f to 0x031840 (0x4fc2 bytes) 000120 1782242253.130923 write 0x03f27d thru 0x03ffff (0xd83 bytes) 000121 1782242253.133855 mapwrite 0x034f05 thru 0x037105 (0x2201 bytes) 000122 1782242253.173599 mapread 0x01dcdf thru 0x02a83e (0xcb60 bytes) 000123 1782242253.178034 read 0x0103e9 thru 0x01865d (0x8275 bytes) 000124 1782242253.179496 read 0x024b27 thru 0x0262ac (0x1786 bytes) 000125 1782242253.181423 fallocate 0x00fd10 thru 0x011fe1 (0x22d2 bytes) 000126 1782242253.190357 punch from 0x013178 to 0x021d2d (0xebb6 bytes) 000127 1782242253.196917 trunc from 0x040000 to 0x03d2f6 (0x2d09 bytes) 000128 1782242253.216439 trunc from 0x03d2f7 to 0x025979 (0x1797d bytes) 000129 1782242253.239512 trunc from 0x016190 to 0x01b642 (0x54b3 bytes) 000130 1782242253.246513 trunc from 0x0043ca to 0x011fe0 (0xdc17 bytes) 000132 1782242253.255207 trunc from 0x004d74 to 0x010b9c (0xbe29 bytes) 000133 1782242253.287250 MIRROR_EXTEND to 2 mirrors 000134 1782242254.337221 trunc from 0x01fcaa to 0x01ffa9 (0x300 bytes) 000135 1782242254.384317 fallocate 0x038404 thru 0x03ffff (0x7bfc bytes) 000136 1782242254.394550 trunc from 0x02597a to 0x02282a (0x314f bytes) 000137 1782242254.439306 write 0x000a48 thru 0x00d1d6 (0xc78f bytes) 000138 1782242254.452101 mapread 0x006614 thru 0x0100cd (0x9aba bytes) 000139 1782242254.483330 mapwrite 0x03a0f8 thru 0x03ffff (0x5f08 bytes) 000140 1782242254.546517 fallocate 0x03f2d8 thru 0x03ffff (0xd28 bytes) 000141 1782242254.553694 write 0x033c4b thru 0x039df3 (0x61a9 bytes) 000142 1782242254.563391 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) 000143 1782242254.777028 punch from 0x027164 to 0x0282f2 (0x118f bytes) 000144 1782242254.785290 read 0x01b37b thru 0x0203ca (0x5050 bytes) 000146 1782242254.803223 trunc from 0x040000 to 0x005ecd (0x3a132 bytes) 000147 1782242254.834447 trunc from 0x00450a to 0x005ecd (0x19c4 bytes) 000148 1782242254.842288 write 0x016831 thru 0x01a6f0 (0x3ec0 bytes) 000150 1782242254.851466 mapwrite 0x0021cc thru 0x0074ca (0x52ff bytes) 000151 1782242254.897629 write 0x027e67 thru 0x028fd2 (0x116c bytes) 000152 1782242254.900226 mapwrite 0x008959 thru 0x00ca50 (0x40f8 bytes) 000153 1782242254.918872 mapread 0x004d8c thru 0x00afbe (0x6233 bytes) 000155 1782242254.921818 punch from 0x003045 to 0x005872 (0x282e bytes) 000156 1782242254.929640 punch from 0x021966 to 0x028fd2 (0x766d bytes) 000157 1782242254.938899 write 0x00e0bd thru 0x01db1c (0xfa60 bytes) 000158 1782242254.950029 read 0x009cfe thru 0x0120f7 (0x83fa bytes) 000159 1782242254.951960 trunc from 0x028fd3 to 0x01ed22 (0xa2b0 bytes) 000160 1782242254.968279 fallocate 0x02b0e1 thru 0x02ee02 (0x3d22 bytes) 000161 1782242254.974184 mapwrite 0x002284 thru 0x009281 (0x6ffe bytes) 000163 1782242255.007847 mapwrite 0x00a8fa thru 0x0160fb (0xb802 bytes) 000164 1782242255.049145 punch from 0x012932 to 0x01ed22 (0xc3f1 bytes) 000165 1782242255.066183 mapread 0x0001ff thru 0x00f1c2 (0xefc4 bytes) 000167 1782242255.070622 mapwrite 0x021f38 thru 0x02a9c2 (0x8a8b bytes) 000168 1782242255.130822 fallocate 0x007c8b thru 0x0178a0 (0xfc16 bytes) 000169 1782242255.145230 trunc from 0x00d2e0 to 0x01a2bf (0xcfe0 bytes) 000170 1782242255.154490 punch from 0x00f1fa to 0x01b6df (0xc4e6 bytes) 000171 1782242255.163711 mapread 0x021c0a thru 0x0261fc (0x45f3 bytes) 000172 1782242255.170175 MIRROR_EXTEND to 2 mirrors 000173 1782242255.896170 mapread 0x00d479 thru 0x011311 (0x3e99 bytes) 000175 1782242255.992614 punch from 0x01c261 to 0x0212d2 (0x5072 bytes) 000176 1782242256.080503 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2a9c3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a9c3): Mirror 2: 0x2eeee310 (stale) CRC-32 checksum value for chunk [0, 0x2a9c3): Mirror 1: 0x2eeee310 Mirror 3: 0x2eeee310 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a9c3: skipped 000177 1782242257.414978 trunc from 0x017e1a to 0x02179c (0x9983 bytes) 000178 1782242257.470700 write 0x026ac1 thru 0x02f11d (0x865d bytes) 000179 1782242257.498652 trunc from 0x01090a to 0x01304d (0x2744 bytes) 000180 1782242257.509008 trunc from 0x02f11e to 0x00385a (0x2b8c3 bytes) 000181 1782242257.547710 trunc from 0x00385b to 0x03c743 (0x38ee9 bytes) 000182 1782242257.570718 fallocate 0x00b71c thru 0x012220 (0x6b05 bytes) 000183 1782242257.584460 fallocate 0x0329e4 thru 0x034340 (0x195d bytes) 000184 1782242257.595279 write 0x0140b8 thru 0x01845e (0x43a7 bytes) 000187 1782242257.620816 fallocate 0x009507 thru 0x00d1a4 (0x3c9e bytes) 000188 1782242257.634367 trunc from 0x00e521 to 0x017978 (0x9458 bytes) 000189 1782242257.668366 punch from 0x0162ea to 0x01e711 (0x8428 bytes) 000190 1782242257.674928 mapwrite 0x036bfe thru 0x038d00 (0x2103 bytes) 000191 1782242257.743441 write 0x02cb24 thru 0x03bb76 (0xf053 bytes) 000192 1782242257.755597 read 0x030bb3 thru 0x03c743 (0xbb91 bytes) 000193 1782242257.756645 read 0x02a4f0 thru 0x030806 (0x6317 bytes) 000194 1782242257.769408 trunc from 0x0181c7 to 0x026946 (0xe780 bytes) 000195 1782242257.776226 read 0x0120f3 thru 0x019623 (0x7531 bytes) 000196 1782242257.800221 mapwrite 0x006928 thru 0x013ade (0xd1b7 bytes) 000197 1782242257.834323 write 0x00ad2b thru 0x00d6fe (0x29d4 bytes) 000198 1782242257.835481 punch from 0x02ed2c to 0x032123 (0x33f8 bytes) 000199 1782242257.843178 fallocate 0x00735c thru 0x011b02 (0xa7a7 bytes) 000200 1782242257.849852 mapread 0x011010 thru 0x015825 (0x4816 bytes) 000201 1782242257.869569 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c744): Mirror 3: 0xf2ecf53f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c744): Mirror 3: 0xf2ecf53f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c744: skipped 000202 1782242258.160513 write 0x032215 thru 0x03b181 (0x8f6d bytes) 000203 1782242258.178406 read 0x025c50 thru 0x02d720 (0x7ad1 bytes) 000204 1782242258.188359 read 0x00d869 thru 0x019d6a (0xc502 bytes) 000205 1782242258.199252 punch from 0x0389d2 to 0x0399e3 (0x1012 bytes) 000206 1782242258.213344 fallocate 0x001240 thru 0x00a099 (0x8e5a bytes) 000207 1782242258.218558 trunc from 0x03c744 to 0x02bf3b (0x10808 bytes) 000208 1782242258.238301 trunc from 0x02bf3c to 0x00d102 (0x1ee39 bytes) 000209 1782242258.259546 write 0x016f36 thru 0x017681 (0x74c bytes) 000210 1782242258.261497 mapwrite 0x00797b thru 0x0150be (0xd744 bytes) 000211 1782242258.348209 read 0x0172b4 thru 0x017681 (0x3ce bytes) 000212 1782242258.349850 trunc from 0x017682 to 0x00885b (0xee26 bytes) 000213 1782242258.399565 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x885c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x885c): Mirror 3: 0xd899c892 (stale) CRC-32 checksum value for chunk [0, 0x885c): Mirror 1: 0x61583b1c Mirror 4: 0x61583b1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x885c: skipped 000214 1782242259.653321 fallocate 0x0378d6 thru 0x03ffff (0x872a bytes) 000215 1782242259.738856 write 0x01cb2c thru 0x02044d (0x3922 bytes) 000216 1782242259.780971 MIRROR_RESYNC 3 mirrors 000217 1782242261.077370 mapread 0x012a02 thru 0x01cbd5 (0xa1d4 bytes) 000218 1782242261.121534 read 0x0360ea thru 0x03ffff (0x9f16 bytes) 000219 1782242261.138186 mapread 0x03fd51 thru 0x03ffff (0x2af bytes) 000220 1782242261.139074 punch from 0x035433 to 0x03ffff (0xabcd bytes) 000221 1782242261.198335 mapread 0x034920 thru 0x035696 (0xd77 bytes) 000222 1782242261.241221 trunc from 0x0060f5 to 0x009d43 (0x3c4f bytes) 000223 1782242261.254342 mapread 0x02ec62 thru 0x0378f7 (0x8c96 bytes) 000224 1782242261.265342 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3b47210d (stale) Mirror 4: 0x3b47210d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2de83fc0 Mirror 5: 0x2de83fc0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000225 1782242262.810775 fallocate 0x033961 thru 0x03b2e0 (0x7980 bytes) 000226 1782242262.903575 read 0x033748 thru 0x03ffff (0xc8b8 bytes) 000227 1782242262.949784 trunc from 0x040000 to 0x0374b9 (0x8b46 bytes) 000228 1782242263.004574 write 0x00e5ac thru 0x00fc2d (0x1682 bytes) 000230 1782242263.022592 fallocate 0x022c91 thru 0x031b0d (0xee7d bytes) 000231 1782242263.045084 read 0x01de47 thru 0x01e4ae (0x668 bytes) 000232 1782242263.074874 fallocate 0x00788b thru 0x01092d (0x90a3 bytes) 000234 1782242263.081787 punch from 0x01121a to 0x014fcc (0x3db3 bytes) 000235 1782242263.093995 write 0x03df2c thru 0x03ffff (0x20d4 bytes) 000236 1782242263.096166 fallocate 0x019267 thru 0x01dd6b (0x4b05 bytes) 000237 1782242263.103489 trunc from 0x040000 to 0x02e105 (0x11efa bytes) 000238 1782242263.138227 read 0x0100ad thru 0x016b9c (0x6af0 bytes) 000239 1782242263.147350 fallocate 0x003292 thru 0x00721c (0x3f8b bytes) 000240 1782242263.172548 write 0x01d850 thru 0x01e5b0 (0xd61 bytes) 000241 1782242263.176515 mapread 0x009bb6 thru 0x0172a4 (0xd6ef bytes) 000242 1782242263.178543 punch from 0x01b199 to 0x023367 (0x81cf bytes) 000243 1782242263.225569 MIRROR_RESYNC 4 mirrors 000244 1782242264.798424 write 0x02346f thru 0x0241f5 (0xd87 bytes) 000245 1782242264.897948 trunc from 0x02e106 to 0x038375 (0xa270 bytes) 000246 1782242264.963515 punch from 0x01094f to 0x01a665 (0x9d17 bytes) 000247 1782242264.998354 mapwrite 0x020a2a thru 0x0218b3 (0xe8a bytes) 000248 1782242265.076155 punch from 0x014e81 to 0x020d25 (0xbea5 bytes) 000249 1782242265.090170 mapwrite 0x02f34a thru 0x039191 (0x9e48 bytes) 000250 1782242265.192657 read 0x01d46f thru 0x02d41c (0xffae bytes) 000251 1782242265.206592 read 0x017b39 thru 0x0251b2 (0xd67a bytes) 000252 1782242265.263967 fallocate 0x010cda thru 0x01f0d3 (0xe3fa bytes) 000253 1782242265.276927 trunc from 0x0205f1 to 0x02c077 (0xba87 bytes) 000254 1782242265.302894 read 0x01b6c5 thru 0x02169d (0x5fd9 bytes) 000255 1782242265.342319 write 0x036cca thru 0x037ab6 (0xded bytes) 000256 1782242265.347257 fallocate 0x00b31e thru 0x011494 (0x6177 bytes) 000257 1782242265.365152 write 0x00c88a thru 0x016312 (0x9a89 bytes) 000258 1782242265.373464 trunc from 0x039192 to 0x001493 (0x37cfe bytes) 000259 1782242265.413757 mapread 0x000d1c thru 0x001493 (0x778 bytes) 000260 1782242265.418491 punch from 0x0004fe to 0x001493 (0xf96 bytes) 000261 1782242265.440758 write 0x02b3b2 thru 0x02ea5f (0x36ae bytes) 000262 1782242265.445481 mapread 0x01b7fd thru 0x01c418 (0xc1c bytes) 000263 1782242265.478464 fallocate 0x00d980 thru 0x017874 (0x9ef5 bytes) 000265 1782242265.488749 trunc from 0x02ea60 to 0x0139f5 (0x1b06a bytes) 000266 1782242265.523643 trunc from 0x006b0b to 0x00861f (0x1b15 bytes) 000267 1782242265.550780 mapwrite 0x00563f thru 0x0142b5 (0xec77 bytes) 000268 1782242265.807172 punch from 0x004956 to 0x00c898 (0x7f43 bytes) 000270 1782242265.831303 write 0x0055ba thru 0x0136d2 (0xe119 bytes) 000272 1782242265.846098 trunc from 0x0142b6 to 0x02d7e2 (0x1952d bytes) 000273 1782242265.879340 punch from 0x014210 to 0x01ee9a (0xac8b bytes) 000274 1782242265.884909 punch from 0x007591 to 0x008d20 (0x1790 bytes) 000275 1782242265.907962 write 0x00a515 thru 0x018e94 (0xe980 bytes) 000276 1782242265.915810 trunc from 0x02d7e3 to 0x006190 (0x27652 bytes) 000277 1782242265.939177 write 0x03bbff thru 0x03ffff (0x4401 bytes) 000278 1782242265.941390 read 0x009eb6 thru 0x017f40 (0xe08b bytes) 000279 1782242265.953911 trunc from 0x040000 to 0x023e79 (0x1c186 bytes) 000280 1782242265.972029 mapread 0x01b526 thru 0x023e79 (0x8954 bytes) 000282 1782242265.975291 trunc from 0x023e7a to 0x009942 (0x1a537 bytes) 000283 1782242265.994033 write 0x0334de thru 0x03caff (0x9622 bytes) 000284 1782242265.997912 trunc from 0x03cb00 to 0x00b2c7 (0x31838 bytes) 000285 1782242266.023775 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb2c8): Mirror 4: 0xa814b623 (stale) Mirror 5: 0xa814b623 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb2c8): Mirror 4: 0xa814b623 (stale) Mirror 5: 0xa814b623 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb2c8: skipped 000286 1782242266.383338 fallocate 0x009b1f thru 0x012db3 (0x9295 bytes) 000287 1782242266.394149 fallocate 0x01baa5 thru 0x028071 (0xc5cd bytes) 000288 1782242266.409992 punch from 0x002c03 to 0x008d97 (0x6195 bytes) 000289 1782242266.423585 mapwrite 0x01cd65 thru 0x021acc (0x4d68 bytes) 000290 1782242266.538507 mapread 0x018a83 thru 0x021acc (0x904a bytes) 000291 1782242266.544766 read 0x017cce thru 0x01c729 (0x4a5c bytes) 000292 1782242266.548402 trunc from 0x021acd to 0x008cdd (0x18def bytes) 000293 1782242266.608983 write 0x0123ea thru 0x017a35 (0x564c bytes) 000294 1782242266.610637 fallocate 0x000ca0 thru 0x0082d8 (0x7639 bytes) 000295 1782242266.621501 mapread 0x00651c thru 0x00ca93 (0x6578 bytes) 000296 1782242266.665324 mapwrite 0x0390f7 thru 0x03b5a1 (0x24ab bytes) 000297 1782242266.751632 trunc from 0x03b5a2 to 0x0026c7 (0x38eda bytes) 000298 1782242266.782407 read 0x000333 thru 0x0026c7 (0x2395 bytes) 000299 1782242266.793990 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26c8): Mirror 5: 0x2e0beb04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26c8): Mirror 5: 0x2e0beb04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26c8: skipped 000300 1782242267.192256 mapread 0x0020d3 thru 0x0026c7 (0x5f5 bytes) 000301 1782242267.205554 fallocate 0x03211a thru 0x03aefb (0x8de2 bytes) 000303 1782242267.219676 trunc from 0x001b50 to 0x0026c7 (0xb78 bytes) 000304 1782242267.235002 trunc from 0x000192 to 0x0026c7 (0x2536 bytes) 000305 1782242267.249395 mapread 0x00163d thru 0x0026c7 (0x108b bytes) 000306 1782242267.285786 read 0x00104b thru 0x0026c7 (0x167d bytes) 000307 1782242267.286817 trunc from 0x001a38 to 0x0026c7 (0xc90 bytes) 000308 1782242267.300337 trunc from 0x0026c8 to 0x03a718 (0x38051 bytes) 000309 1782242267.387319 trunc from 0x03a719 to 0x0304a3 (0xa275 bytes) 000310 1782242267.445747 fallocate 0x00a2b9 thru 0x015f62 (0xbcaa bytes) 000311 1782242267.452298 fallocate 0x03b8db thru 0x03ffff (0x4725 bytes) 000314 1782242267.465371 write 0x013e09 thru 0x0237e4 (0xf9dc bytes) 000315 1782242267.497108 mapread 0x006c41 thru 0x014e47 (0xe207 bytes) 000317 1782242267.522153 trunc from 0x0304a4 to 0x00a024 (0x2647f bytes) 000318 1782242267.544476 trunc from 0x001118 to 0x00a024 (0x8f0d bytes) 000319 1782242267.557133 fallocate 0x013419 thru 0x020c0c (0xd7f4 bytes) 000320 1782242267.584780 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) 000321 1782242267.878354 write 0x01e4b6 thru 0x0286bd (0xa208 bytes) 000322 1782242267.903673 mapwrite 0x03c798 thru 0x03d2b2 (0xb1b bytes) 000325 1782242268.001218 read 0x01cee7 thru 0x024560 (0x767a bytes) 000326 1782242268.021747 MIRROR_EXTEND to 2 mirrors 000327 1782242268.922219 trunc from 0x01af79 to 0x029ce8 (0xed70 bytes) 000328 1782242268.959168 trunc from 0x008cb2 to 0x00c25a (0x35a9 bytes) 000329 1782242268.967910 punch from 0x01da0c to 0x02aa5a (0xd04f bytes) 000331 1782242268.978784 fallocate 0x02eac3 thru 0x03ac9e (0xc1dc bytes) 000333 1782242268.992622 read 0x03c459 thru 0x03d2b2 (0xe5a bytes) 000334 1782242269.000467 punch from 0x015f05 to 0x01f928 (0x9a24 bytes) 000335 1782242269.021726 fallocate 0x039ed4 thru 0x03a3cc (0x4f9 bytes) 000336 1782242269.028854 read 0x007de8 thru 0x00fd7d (0x7f96 bytes) 000338 1782242269.058307 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d2b3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d2b3): Mirror 2: 0x1185b0dd (stale) CRC-32 checksum value for chunk [0, 0x3d2b3): Mirror 1: 0x652248ff Mirror 3: 0x652248ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d2b3: skipped 000339 1782242270.336279 punch from 0x027c0e to 0x034cf6 (0xd0e9 bytes) 000340 1782242270.402649 mapwrite 0x010364 thru 0x01ccf5 (0xc992 bytes) 000341 1782242270.518562 trunc from 0x03d2b3 to 0x03cba4 (0x70e bytes) 000342 1782242270.568544 mapread 0x037604 thru 0x03cba4 (0x55a1 bytes) 000343 1782242270.608588 fallocate 0x00d98e thru 0x0120de (0x4751 bytes) 000344 1782242270.614841 read 0x006638 thru 0x0153b5 (0xed7e bytes) 000345 1782242270.636495 punch from 0x02cd7f to 0x034853 (0x7ad5 bytes) 000346 1782242270.644733 trunc from 0x03cba5 to 0x01b01e (0x21b86 bytes) 000347 1782242270.677167 write 0x005a02 thru 0x009a12 (0x4011 bytes) 000348 1782242270.707075 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b01f): Mirror 2: 0x1f41236b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b01f): Mirror 2: 0x1f41236b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b01f: skipped 000349 1782242271.165118 mapread 0x012e35 thru 0x01a14f (0x731b bytes) 000350 1782242271.202433 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) 000351 1782242271.374568 fallocate 0x02b501 thru 0x02c162 (0xc62 bytes) 000353 1782242271.406283 trunc from 0x02c163 to 0x036e0e (0xacac bytes) 000354 1782242271.449978 read 0x007eea thru 0x00ddcd (0x5ee4 bytes) 000355 1782242271.466527 punch from 0x017ea1 to 0x01a105 (0x2265 bytes) 000356 1782242271.473774 mapwrite 0x036f82 thru 0x03f435 (0x84b4 bytes) 000357 1782242271.547731 trunc from 0x02a306 to 0x033d08 (0x9a03 bytes) 000359 1782242271.558934 read 0x036bf5 thru 0x03f435 (0x8841 bytes) 000360 1782242271.571225 MIRROR_EXTEND to 2 mirrors 000361 1782242272.623086 fallocate 0x0054a4 thru 0x0139bd (0xe51a bytes) 000362 1782242272.668769 read 0x00ba38 thru 0x00d996 (0x1f5f bytes) 000363 1782242272.680760 mapwrite 0x01f45b thru 0x02a452 (0xaff8 bytes) 000364 1782242272.780386 write 0x00757c thru 0x00bc78 (0x46fd bytes) 000366 1782242272.804990 punch from 0x03421f to 0x03778e (0x3570 bytes) 000367 1782242272.811942 punch from 0x018861 to 0x01d288 (0x4a28 bytes) 000368 1782242272.817620 punch from 0x02e357 to 0x03e347 (0xfff1 bytes) 000369 1782242272.823243 write 0x00242b thru 0x00fe2e (0xda04 bytes) 000370 1782242272.834991 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f436): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f436): Mirror 2: 0x5b2c58a (stale) CRC-32 checksum value for chunk [0, 0x3f436): Mirror 1: 0x8d2aac92 Mirror 3: 0x8d2aac92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f436: skipped 000371 1782242274.167905 punch from 0x02a5b8 to 0x033249 (0x8c92 bytes) 000372 1782242274.246364 trunc from 0x001d28 to 0x0029a1 (0xc7a bytes) 000374 1782242274.256776 punch from 0x015818 to 0x01c116 (0x68ff bytes) 000375 1782242274.267009 read 0x015013 thru 0x023439 (0xe427 bytes) 000376 1782242274.278676 fallocate 0x01cab4 thru 0x025140 (0x868d bytes) 000377 1782242274.285329 mapwrite 0x02e25a thru 0x02f945 (0x16ec bytes) 000378 1782242274.308693 write 0x001acd thru 0x0066db (0x4c0f bytes) 000379 1782242274.327326 read 0x005e4a thru 0x00ef2d (0x90e4 bytes) 000381 1782242274.339742 write 0x032513 thru 0x039ae0 (0x75ce bytes) 000382 1782242274.341196 read 0x0360cf thru 0x03f435 (0x9367 bytes) 000383 1782242274.344309 trunc from 0x00daf2 to 0x014d09 (0x7218 bytes) 000384 1782242274.356462 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f436): Mirror 2: 0x5b2c58a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f436): Mirror 2: 0x5b2c58a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f436: skipped 000385 1782242274.765112 fallocate 0x0350de thru 0x03db65 (0x8a88 bytes) 000386 1782242274.780517 fallocate 0x00e724 thru 0x014464 (0x5d41 bytes) 000388 1782242274.793664 write 0x02671a thru 0x035126 (0xea0d bytes) 000389 1782242274.816387 mapread 0x01301f thru 0x021d29 (0xed0b bytes) 000391 1782242274.852937 fallocate 0x01c41e thru 0x0246f2 (0x82d5 bytes) 000393 1782242274.860227 write 0x0261f8 thru 0x026b29 (0x932 bytes) 000394 1782242274.861883 trunc from 0x007dd2 to 0x00c75d (0x498c bytes) 000395 1782242274.866868 write 0x001fc2 thru 0x011ef4 (0xff33 bytes) 000396 1782242274.877346 write 0x037600 thru 0x03ffff (0x8a00 bytes) 000397 1782242274.884736 trunc from 0x03920e to 0x03ffff (0x6df2 bytes) 000399 1782242274.928702 mapread 0x014a79 thru 0x016069 (0x15f1 bytes) 000400 1782242274.958698 MIRROR_RESYNC 2 mirrors 000401 1782242275.960122 mapwrite 0x020476 thru 0x026fe1 (0x6b6c bytes) 000402 1782242276.094723 mapread 0x00d719 thru 0x01bf6c (0xe854 bytes) 000403 1782242276.139200 mapread 0x02edfc thru 0x03ea28 (0xfc2d bytes) 000404 1782242276.176956 read 0x00994a thru 0x009a66 (0x11d bytes) 000405 1782242276.179711 write 0x03bb78 thru 0x03e19c (0x2625 bytes) 000406 1782242276.181666 trunc from 0x03ed19 to 0x03ffff (0x12e7 bytes) 000407 1782242276.205785 punch from 0x00ceb5 to 0x01a0ae (0xd1fa bytes) 000408 1782242276.218827 trunc from 0x03135e to 0x03ef61 (0xdc04 bytes) 000409 1782242276.228362 write 0x00c0c3 thru 0x0134bb (0x73f9 bytes) 000410 1782242276.242906 mapread 0x00e574 thru 0x017467 (0x8ef4 bytes) 000411 1782242276.286364 fallocate 0x001c1e thru 0x00a7d6 (0x8bb9 bytes) 000412 1782242276.292579 mapwrite 0x02c1cf thru 0x0337ce (0x7600 bytes) 000413 1782242276.325651 write 0x001c80 thru 0x00394b (0x1ccc bytes) 000414 1782242276.337969 trunc from 0x0284e4 to 0x0319b7 (0x94d4 bytes) 000415 1782242276.343736 trunc from 0x0391f8 to 0x03e096 (0x4e9f bytes) 000416 1782242276.354733 mapread 0x003d8f thru 0x006e35 (0x30a7 bytes) 000417 1782242276.374623 MIRROR_RESYNC 2 mirrors 000418 1782242277.306557 fallocate 0x0169b4 thru 0x01edf5 (0x8442 bytes) 000419 1782242277.372584 trunc from 0x032d45 to 0x0385b6 (0x5872 bytes) 000420 1782242277.382789 mapread 0x036152 thru 0x03ffff (0x9eae bytes) 000421 1782242277.403325 mapwrite 0x0222df thru 0x02b74d (0x946f bytes) 000423 1782242277.442630 mapwrite 0x0262b6 thru 0x0342f2 (0xe03d bytes) 000424 1782242277.472249 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) 000425 1782242277.666900 fallocate 0x00da01 thru 0x017d3e (0xa33e bytes) 000428 1782242277.676658 mapwrite 0x012335 thru 0x019215 (0x6ee1 bytes) 000430 1782242277.771132 fallocate 0x03c1b4 thru 0x03ee61 (0x2cae bytes) 000431 1782242277.787673 fallocate 0x025f37 thru 0x02e47c (0x8546 bytes) 000432 1782242277.800331 mapwrite 0x03d165 thru 0x03ffff (0x2e9b bytes) 000434 1782242277.844714 write 0x0156da thru 0x018056 (0x297d bytes) 000435 1782242277.856266 write 0x033ddf thru 0x03ffff (0xc221 bytes) 000436 1782242277.872731 fallocate 0x0074a6 thru 0x00fabb (0x8616 bytes) 000437 1782242277.879685 trunc from 0x0318a8 to 0x03ffff (0xe758 bytes) 000438 1782242277.898175 trunc from 0x03233e to 0x034be3 (0x28a6 bytes) 000439 1782242277.913025 mapwrite 0x031d84 thru 0x03415a (0x23d7 bytes) 000440 1782242277.984630 fallocate 0x027137 thru 0x03006d (0x8f37 bytes) 000441 1782242278.006467 mapread 0x031889 thru 0x03923f (0x79b7 bytes) 000442 1782242278.014550 trunc from 0x03714b to 0x03ffff (0x8eb5 bytes) 000444 1782242278.039775 punch from 0x03be2c to 0x03ffff (0x41d4 bytes) 000445 1782242278.048356 trunc from 0x03655a to 0x03b117 (0x4bbe bytes) 000446 1782242278.059212 mapwrite 0x00bb64 thru 0x012bb7 (0x7054 bytes) 000447 1782242278.101505 mapwrite 0x0350d4 thru 0x03f3b7 (0xa2e4 bytes) 000448 1782242278.207739 mapread 0x00f7ea thru 0x012a07 (0x321e bytes) 000449 1782242278.218382 read 0x03006f thru 0x03b584 (0xb516 bytes) 000450 1782242278.222553 trunc from 0x00bd10 to 0x00e168 (0x2459 bytes) 000451 1782242278.241446 punch from 0x0399f3 to 0x03ffff (0x660d bytes) 000452 1782242278.267819 mapread 0x028518 thru 0x034386 (0xbe6f bytes) 000455 1782242278.309562 mapread 0x02d61b thru 0x032665 (0x504b bytes) 000456 1782242278.314396 punch from 0x012dbc to 0x022d33 (0xff78 bytes) 000457 1782242278.330424 read 0x0015c0 thru 0x004a21 (0x3462 bytes) 000459 1782242278.361687 read 0x024969 thru 0x028d7c (0x4414 bytes) 000460 1782242278.364467 read 0x002268 thru 0x00c1ce (0x9f67 bytes) 000461 1782242278.366212 mapread 0x020ea0 thru 0x029e8f (0x8ff0 bytes) 000462 1782242278.377416 mapwrite 0x014250 thru 0x01acc4 (0x6a75 bytes) 000463 1782242278.435967 punch from 0x017964 to 0x022911 (0xafae bytes) 000464 1782242278.444125 trunc from 0x040000 to 0x01653d (0x29ac2 bytes) 000465 1782242278.518338 mapread 0x015dfa thru 0x01653d (0x744 bytes) 000466 1782242278.519502 read 0x013e6f thru 0x01653d (0x26cf bytes) 000468 1782242278.526799 mapwrite 0x02ff04 thru 0x03dd17 (0xde14 bytes) 000469 1782242278.643443 mapread 0x03c8bc thru 0x03dd17 (0x145c bytes) 000471 1782242278.648352 read 0x02602d thru 0x035f85 (0xff59 bytes) 000472 1782242278.649292 trunc from 0x0025d9 to 0x0051e7 (0x2c0f bytes) 000473 1782242278.657022 mapread 0x02b54b thru 0x02f9cd (0x4483 bytes) 000475 1782242278.660661 fallocate 0x01feba thru 0x02cf39 (0xd080 bytes) 000476 1782242278.667388 fallocate 0x003697 thru 0x0062fd (0x2c67 bytes) 000478 1782242278.675180 trunc from 0x021d89 to 0x027925 (0x5b9d bytes) 000479 1782242278.681283 trunc from 0x03dd18 to 0x01af14 (0x22e03 bytes) 000480 1782242278.705289 write 0x029b29 thru 0x036406 (0xc8de bytes) 000481 1782242278.710374 mapread 0x027de4 thru 0x0356d7 (0xd8f4 bytes) 000483 1782242278.737413 trunc from 0x036407 to 0x014850 (0x21bb6 bytes) 000486 1782242278.792673 write 0x0071a7 thru 0x00c74c (0x55a6 bytes) 000487 1782242278.795676 trunc from 0x014851 to 0x002ca9 (0x11ba7 bytes) 000488 1782242278.827013 mapwrite 0x012526 thru 0x020daa (0xe885 bytes) 000489 1782242279.007302 punch from 0x01c15f to 0x020daa (0x4c4c bytes) 000492 1782242279.024888 fallocate 0x03e975 thru 0x03ffff (0x168b bytes) 000493 1782242279.030176 write 0x03128f thru 0x03f095 (0xde07 bytes) 000494 1782242279.044382 trunc from 0x02d992 to 0x036572 (0x8be1 bytes) 000495 1782242279.063209 fallocate 0x008d6e thru 0x018592 (0xf825 bytes) 000496 1782242279.069297 trunc from 0x040000 to 0x0297fc (0x16803 bytes) 000497 1782242279.088299 mapread 0x01ac8a thru 0x01cd5c (0x20d3 bytes) 000498 1782242279.094294 trunc from 0x0297fd to 0x020d22 (0x8ada bytes) 000499 1782242279.111400 read 0x00aa1e thru 0x00cbf4 (0x21d7 bytes) 000500 1782242279.123889 punch from 0x01597a to 0x015e54 (0x4db bytes) 000501 1782242279.132314 fallocate 0x037d50 thru 0x03ec03 (0x6eb4 bytes) 000502 1782242279.137731 mapwrite 0x026101 thru 0x02fc4b (0x9b4b bytes) 000503 1782242279.200433 trunc from 0x01a9f2 to 0x02482f (0x9e3e bytes) 000505 1782242279.223309 write 0x0137b1 thru 0x01b874 (0x80c4 bytes) 000506 1782242279.235447 trunc from 0x02386a to 0x02432d (0xac4 bytes) 000507 1782242279.243547 read 0x02e375 thru 0x02fc4b (0x18d7 bytes) 000508 1782242279.244718 mapread 0x019502 thru 0x01aa06 (0x1505 bytes) 000509 1782242279.246653 punch from 0x0117e7 to 0x02089d (0xf0b7 bytes) 000510 1782242279.279533 read 0x01a597 thru 0x026d70 (0xc7da bytes) 000512 1782242279.297794 mapwrite 0x02ebd4 thru 0x03a3ad (0xb7da bytes) 000513 1782242279.395400 fallocate 0x0183cf thru 0x02166f (0x92a1 bytes) 000514 1782242279.405797 fallocate 0x02db26 thru 0x031f06 (0x43e1 bytes) 000515 1782242279.423768 punch from 0x02b7f9 to 0x02b8c1 (0xc9 bytes) 000516 1782242279.440293 trunc from 0x027c26 to 0x02cf76 (0x5351 bytes) 000518 1782242279.460011 punch from 0x0060ce to 0x00a1b2 (0x40e5 bytes) 000520 1782242279.472441 mapwrite 0x0276a8 thru 0x027ee8 (0x841 bytes) 000522 1782242279.494172 trunc from 0x03a3ae to 0x016fac (0x23401 bytes) 000523 1782242279.509845 trunc from 0x016fad to 0x001597 (0x15a15 bytes) 000525 1782242279.546947 mapread 0x000984 thru 0x001597 (0xc14 bytes) 000526 1782242279.548435 trunc from 0x000d24 to 0x001597 (0x874 bytes) 000527 1782242279.564960 mapread 0x000556 thru 0x001597 (0x1042 bytes) 000528 1782242279.597678 write 0x034cc5 thru 0x03ffff (0xb33b bytes) 000529 1782242279.600985 mapwrite 0x02e985 thru 0x03ba49 (0xd0c5 bytes) 000530 1782242279.672184 fallocate 0x004578 thru 0x006118 (0x1ba1 bytes) 000531 1782242279.682497 punch from 0x020023 to 0x020f54 (0xf32 bytes) 000533 1782242279.695999 fallocate 0x03e8df thru 0x03ffff (0x1721 bytes) 000534 1782242279.709325 fallocate 0x01789b thru 0x0257a9 (0xdf0f bytes) 000535 1782242279.726125 punch from 0x00ee14 to 0x0162b2 (0x749f bytes) 000536 1782242279.734624 mapread 0x03ad8c thru 0x03ffff (0x5274 bytes) 000538 1782242279.751604 MIRROR_EXTEND to 2 mirrors 000539 1782242280.661831 read 0x03912b thru 0x03ffff (0x6ed5 bytes) 000540 1782242280.693210 mapwrite 0x03001a thru 0x035013 (0x4ffa bytes) 000541 1782242280.833200 read 0x011e4e thru 0x01757e (0x5731 bytes) 000542 1782242280.843998 trunc from 0x040000 to 0x0326a7 (0xd958 bytes) 000544 1782242280.891417 fallocate 0x03705c thru 0x03ffff (0x8fa4 bytes) 000545 1782242280.899193 mapwrite 0x006bb6 thru 0x012912 (0xbd5d bytes) 000546 1782242280.926222 MIRROR_RESYNC 2 mirrors 000547 1782242281.760031 write 0x0352cc thru 0x03ffff (0xad34 bytes) 000548 1782242281.807412 mapread 0x00021b thru 0x002bc5 (0x29ab bytes) 000549 1782242281.837002 mapread 0x006f93 thru 0x011f8e (0xaffc bytes) 000550 1782242281.861447 write 0x02fe11 thru 0x0399c7 (0x9bb7 bytes) 000551 1782242281.869710 trunc from 0x040000 to 0x01b523 (0x24adc bytes) 000552 1782242281.898991 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) 000553 1782242282.061985 MIRROR_EXTEND to 2 mirrors 000554 1782242283.115970 read 0x002a48 thru 0x00b14b (0x8704 bytes) 000555 1782242283.160817 punch from 0x00a4a8 to 0x00bca2 (0x17fb bytes) 000556 1782242283.198169 trunc from 0x01b524 to 0x035046 (0x19b23 bytes) 000557 1782242283.243005 write 0x025961 thru 0x02d86f (0x7f0f bytes) 000558 1782242283.255974 fallocate 0x03d79c thru 0x03ffff (0x2864 bytes) 000559 1782242283.261722 fallocate 0x026cf0 thru 0x029791 (0x2aa2 bytes) 000560 1782242283.267598 write 0x004f53 thru 0x007c4e (0x2cfc bytes) 000561 1782242283.278228 trunc from 0x035047 to 0x02456a (0x10adc bytes) 000562 1782242283.299362 trunc from 0x014210 to 0x022d72 (0xeb63 bytes) 000563 1782242283.311716 mapwrite 0x03d19c thru 0x03ffff (0x2e64 bytes) 000564 1782242283.396870 read 0x00d050 thru 0x01a7ed (0xd79e bytes) 000565 1782242283.412968 trunc from 0x040000 to 0x002858 (0x3d7a7 bytes) 000566 1782242283.446959 read 0x000c2c thru 0x002858 (0x1c2d bytes) 000567 1782242283.453633 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2859): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2859): Mirror 2: 0xc1b82bad (stale) CRC-32 checksum value for chunk [0, 0x2859): Mirror 1: 0xc1b82bad Mirror 3: 0xc1b82bad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2859: skipped 000568 1782242284.576682 fallocate 0x020a97 thru 0x028b4b (0x80b5 bytes) 000569 1782242284.622275 fallocate 0x0043f6 thru 0x00c4c7 (0x80d2 bytes) 000571 1782242284.630179 punch from 0x005f3d to 0x00c4c7 (0x658b bytes) 000572 1782242284.635937 mapread 0x00251c thru 0x00c4c7 (0x9fac bytes) 000573 1782242284.664243 punch from 0x00937e to 0x00c4c7 (0x314a bytes) 000574 1782242284.669698 mapwrite 0x00ab6f thru 0x00eccf (0x4161 bytes) 000575 1782242284.773678 mapread 0x00b475 thru 0x00eccf (0x385b bytes) 000576 1782242284.774909 mapwrite 0x023657 thru 0x0335ac (0xff56 bytes) 000577 1782242284.937266 trunc from 0x0335ad to 0x03fc90 (0xc6e4 bytes) 000578 1782242284.964364 trunc from 0x02c4ad to 0x03bf39 (0xfa8d bytes) 000579 1782242284.974240 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fc91): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc91): Mirror 2: 0xce54c011 (stale) Mirror 3: 0x9237d7b8 (stale) CRC-32 checksum value for chunk [0, 0x3fc91): Mirror 1: 0x441c87b6 Mirror 4: 0x441c87b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc91: skipped 000580 1782242286.287781 punch from 0x0180ce to 0x02001c (0x7f4f bytes) 000582 1782242286.365723 mapwrite 0x0335e0 thru 0x03e2b4 (0xacd5 bytes) 000583 1782242286.418675 read 0x00803a thru 0x008f9d (0xf64 bytes) 000584 1782242286.429360 punch from 0x020f0e to 0x027fc3 (0x70b6 bytes) 000585 1782242286.436167 mapread 0x02aa9a thru 0x0362d8 (0xb83f bytes) 000586 1782242286.454208 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fc91): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc91): Mirror 2: 0xce54c011 (stale) Mirror 3: 0x9237d7b8 (stale) Mirror 4: 0x441c87b6 (stale) CRC-32 checksum value for chunk [0, 0x3fc91): Mirror 1: 0x27f87312 Mirror 5: 0x27f87312 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc91: skipped 000587 1782242287.537690 write 0x01ed87 thru 0x020418 (0x1692 bytes) 000588 1782242287.620318 trunc from 0x03fc91 to 0x014f9f (0x2acf1 bytes) 000589 1782242287.673314 read 0x001889 thru 0x00a25d (0x89d5 bytes) 000590 1782242287.684913 write 0x025732 thru 0x02c909 (0x71d8 bytes) 000591 1782242287.687479 write 0x02227b thru 0x02b532 (0x92b8 bytes) 000592 1782242287.699933 write 0x037639 thru 0x0379c5 (0x38d bytes) 000593 1782242287.703963 fallocate 0x030eb4 thru 0x034aa7 (0x3bf4 bytes) 000594 1782242287.709945 trunc from 0x036075 to 0x0379c5 (0x1951 bytes) 000595 1782242287.727936 punch from 0x0165e6 to 0x01d1a3 (0x6bbe bytes) 000596 1782242287.733978 punch from 0x02a64c to 0x02aeba (0x86f bytes) 000597 1782242287.739302 trunc from 0x02e2b7 to 0x0335db (0x5325 bytes) 000598 1782242287.746193 fallocate 0x01d543 thru 0x028fad (0xba6b bytes) 000599 1782242287.753296 write 0x02bb9e thru 0x033303 (0x7766 bytes) 000600 1782242287.760031 write 0x01e3a7 thru 0x02d5da (0xf234 bytes) 000601 1782242287.768936 mapread 0x025d06 thru 0x033242 (0xd53d bytes) 000602 1782242287.778264 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x379c6): Mirror 2: 0xce7de26e (stale) Mirror 4: 0xbdb81275 (stale) Mirror 5: 0xa702e06c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x379c6): Mirror 2: 0xce7de26e (stale) Mirror 4: 0xbdb81275 (stale) Mirror 5: 0xa702e06c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x379c6: skipped 000605 1782242288.335201 mapwrite 0x0304f3 thru 0x037227 (0x6d35 bytes) 000606 1782242288.449217 trunc from 0x0379c6 to 0x03c8a5 (0x4ee0 bytes) 000607 1782242288.496610 mapwrite 0x006c38 thru 0x00df28 (0x72f1 bytes) 000608 1782242288.555498 trunc from 0x02e6fb to 0x033075 (0x497b bytes) 000609 1782242288.563416 write 0x02fff2 thru 0x0320d9 (0x20e8 bytes) 000610 1782242288.581003 punch from 0x0150c3 to 0x01824b (0x3189 bytes) 000611 1782242288.586121 trunc from 0x01b765 to 0x0253d6 (0x9c72 bytes) 000612 1782242288.591924 mapread 0x01027b thru 0x0111e6 (0xf6c bytes) 000613 1782242288.604737 mapread 0x03097a thru 0x03332c (0x29b3 bytes) 000614 1782242288.616392 punch from 0x0380bf to 0x038328 (0x26a bytes) 000615 1782242288.622209 fallocate 0x022ae5 thru 0x02f13e (0xc65a bytes) 000616 1782242288.634382 MIRROR_RESYNC 4 mirrors 000618 1782242290.463707 read 0x01ed8d thru 0x021e72 (0x30e6 bytes) 000619 1782242290.475631 mapwrite 0x0342e5 thru 0x0344a0 (0x1bc bytes) 000620 1782242290.566008 punch from 0x006751 to 0x00fb87 (0x9437 bytes) 000621 1782242290.582740 mapwrite 0x033e56 thru 0x03c135 (0x82e0 bytes) 000622 1782242290.625607 write 0x007e9a thru 0x0116f5 (0x985c bytes) 000623 1782242290.650332 mapwrite 0x010caf thru 0x0198f4 (0x8c46 bytes) 000624 1782242290.677965 read 0x031b11 thru 0x0387f7 (0x6ce7 bytes) 000625 1782242290.680198 trunc from 0x03c8a6 to 0x0196e1 (0x231c4 bytes) 000626 1782242290.729826 mapread 0x00b21d thru 0x018f63 (0xdd47 bytes) 000628 1782242290.743160 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x196e2): Mirror 2: 0x4405cc04 (stale) Mirror 4: 0x4405cc04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x196e2): Mirror 2: 0x4405cc04 (stale) Mirror 4: 0x4405cc04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x196e2: skipped 000629 1782242291.227238 mapwrite 0x028633 thru 0x0338b4 (0xb282 bytes) 000630 1782242291.303246 trunc from 0x02146e to 0x0230de (0x1c71 bytes) 000632 1782242291.320651 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x338b5): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x338b5): Mirror 2: 0xdf882085 (stale) Mirror 4: 0xdf882085 (stale) CRC-32 checksum value for chunk [0, 0x338b5): Mirror 1: 0x9b70536e Mirror 5: 0x9b70536e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x338b5: skipped 000633 1782242292.490010 fallocate 0x02eb0c thru 0x031531 (0x2a26 bytes) 000634 1782242292.541473 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x338b5): Mirror 2: 0xdf882085 (stale) Mirror 5: 0x9b70536e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x338b5): Mirror 2: 0xdf882085 (stale) Mirror 5: 0x9b70536e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x338b5: skipped 000636 1782242293.007613 fallocate 0x00e970 thru 0x0162c2 (0x7953 bytes) 000637 1782242293.026449 trunc from 0x0338b5 to 0x0248da (0xefda bytes) 000638 1782242293.120187 read 0x01f3ae thru 0x02090b (0x155e bytes) 000640 1782242293.134946 punch from 0x009a05 to 0x00e143 (0x473f bytes) 000641 1782242293.145745 MIRROR_RESYNC 3 mirrors 000642 1782242294.503089 fallocate 0x017f97 thru 0x025710 (0xd77a bytes) 000644 1782242294.573178 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25711): Mirror 2: 0xc3421d6d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25711): Mirror 2: 0xc3421d6d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25711: skipped 000645 1782242294.908825 trunc from 0x025711 to 0x03e52e (0x18e1e bytes) 000646 1782242294.946746 trunc from 0x025403 to 0x02dc41 (0x883f bytes) 000647 1782242294.961271 mapread 0x000b68 thru 0x008c85 (0x811e bytes) 000648 1782242295.010018 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) 000649 1782242295.326996 write 0x020a0f thru 0x029647 (0x8c39 bytes) 000650 1782242295.368681 punch from 0x001879 to 0x00ab90 (0x9318 bytes) 000651 1782242295.378285 trunc from 0x03e52f to 0x008b98 (0x35996 bytes) 000652 1782242295.404245 mapread 0x0028bf thru 0x006724 (0x3e66 bytes) 000653 1782242295.438021 mapread 0x00293f thru 0x005909 (0x2fcb bytes) 000654 1782242295.440524 read 0x0078f7 thru 0x008b98 (0x12a2 bytes) 000655 1782242295.445332 trunc from 0x000881 to 0x008b98 (0x8318 bytes) 000656 1782242295.450617 fallocate 0x003c23 thru 0x012136 (0xe514 bytes) 000658 1782242295.456877 read 0x009d7a thru 0x00d46e (0x36f5 bytes) 000659 1782242295.466159 read 0x011224 thru 0x012136 (0xf13 bytes) 000660 1782242295.467894 read 0x002e86 thru 0x00b16b (0x82e6 bytes) 000661 1782242295.468677 trunc from 0x012137 to 0x02c3e3 (0x1a2ad bytes) 000662 1782242295.490566 mapread 0x01535d thru 0x018d71 (0x3a15 bytes) 000663 1782242295.513361 mapwrite 0x02d98f thru 0x039440 (0xbab2 bytes) 000665 1782242295.603626 write 0x01b2ec thru 0x01cbc8 (0x18dd bytes) 000666 1782242295.604854 mapwrite 0x0271b3 thru 0x0287c9 (0x1617 bytes) 000667 1782242295.654793 fallocate 0x014e20 thru 0x020af9 (0xbcda bytes) 000668 1782242295.669030 fallocate 0x02253c thru 0x030aa3 (0xe568 bytes) 000669 1782242295.686498 mapread 0x02303a thru 0x024be9 (0x1bb0 bytes) 000670 1782242295.698987 read 0x0332ee thru 0x039440 (0x6153 bytes) 000672 1782242295.703104 mapread 0x037ce8 thru 0x039440 (0x1759 bytes) 000673 1782242295.705184 write 0x03b002 thru 0x03ffff (0x4ffe bytes) 000674 1782242295.708435 mapwrite 0x00e65e thru 0x016499 (0x7e3c bytes) 000675 1782242295.770009 MIRROR_EXTEND to 2 mirrors 000676 1782242296.737010 trunc from 0x040000 to 0x0165e1 (0x29a1e bytes) 000677 1782242296.824535 trunc from 0x0029ef to 0x00321c (0x82e bytes) 000679 1782242296.831005 mapwrite 0x023235 thru 0x02bacd (0x8899 bytes) 000680 1782242297.018923 MIRROR_RESYNC 2 mirrors 000681 1782242297.748258 punch from 0x01cd55 to 0x02bacd (0xed79 bytes) 000684 1782242297.774435 trunc from 0x02bace to 0x028d9c (0x2d31 bytes) 000685 1782242297.813194 read 0x025117 thru 0x028d9c (0x3c86 bytes) 000686 1782242297.824574 trunc from 0x019524 to 0x019778 (0x255 bytes) 000687 1782242297.831075 trunc from 0x0192f3 to 0x02753d (0xe24b bytes) 000688 1782242297.842137 read 0x004f57 thru 0x01259b (0xd645 bytes) 000689 1782242297.857398 read 0x025918 thru 0x028d9c (0x3485 bytes) 000690 1782242297.859252 read 0x020b24 thru 0x022c58 (0x2135 bytes) 000691 1782242297.860593 mapwrite 0x03dfc7 thru 0x03ffff (0x2039 bytes) 000692 1782242297.965391 fallocate 0x020190 thru 0x02fe36 (0xfca7 bytes) 000693 1782242297.992778 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd5fa1a94 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xca557f22 Mirror 3: 0xca557f22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000694 1782242299.272218 write 0x00cb87 thru 0x01900b (0xc485 bytes) 000695 1782242299.345756 trunc from 0x040000 to 0x0344ac (0xbb53 bytes) 000696 1782242299.401988 fallocate 0x02045c thru 0x02b52e (0xb0d3 bytes) 000697 1782242299.411596 trunc from 0x0146bf to 0x01d364 (0x8ca6 bytes) 000698 1782242299.434842 mapread 0x0083bd thru 0x010ca0 (0x88e4 bytes) 000699 1782242299.468164 mapread 0x02d54f thru 0x0335d0 (0x6082 bytes) 000702 1782242299.485576 mapwrite 0x03b8ef thru 0x03ffff (0x4711 bytes) 000703 1782242299.529872 mapwrite 0x006c46 thru 0x008546 (0x1901 bytes) 000704 1782242299.543360 trunc from 0x02c66e to 0x02d2d8 (0xc6b bytes) 000705 1782242299.549538 mapread 0x01f01a thru 0x02ece9 (0xfcd0 bytes) 000707 1782242299.569942 punch from 0x022b8b to 0x031d6c (0xf1e2 bytes) 000708 1782242299.577868 trunc from 0x040000 to 0x01fe98 (0x20167 bytes) 000709 1782242299.614373 trunc from 0x01fe99 to 0x020755 (0x8bd bytes) 000710 1782242299.635170 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x20756): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20756): Mirror 2: 0x87e094e8 (stale) Mirror 3: 0x87e094e8 (stale) CRC-32 checksum value for chunk [0, 0x20756): Mirror 1: 0x29af1865 Mirror 4: 0x29af1865 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20756: skipped 000711 1782242300.961528 MIRROR_RESYNC 4 mirrors 000712 1782242302.451411 punch from 0x019c61 to 0x01aaec (0xe8c bytes) 000713 1782242302.499124 mapread 0x008ab3 thru 0x011e9c (0x93ea bytes) 000714 1782242302.520715 read 0x0035a7 thru 0x00a76b (0x71c5 bytes) 000715 1782242302.521874 trunc from 0x020756 to 0x039f4e (0x197f9 bytes) 000716 1782242302.552767 read 0x007113 thru 0x0116a5 (0xa593 bytes) 000717 1782242302.553629 punch from 0x020314 to 0x0255a8 (0x5295 bytes) 000718 1782242302.560435 punch from 0x03875b to 0x039f4e (0x17f4 bytes) 000719 1782242302.565493 read 0x00c6e9 thru 0x01c44a (0xfd62 bytes) 000720 1782242302.577949 mapwrite 0x03970b thru 0x03cc14 (0x350a bytes) 000721 1782242302.640664 punch from 0x00bc2b to 0x00e5e9 (0x29bf bytes) 000722 1782242302.646859 trunc from 0x03cc15 to 0x03d076 (0x462 bytes) 000723 1782242302.665763 read 0x01bc13 thru 0x020d30 (0x511e bytes) 000724 1782242302.667111 write 0x03a459 thru 0x03ffff (0x5ba7 bytes) 000725 1782242302.674730 MIRROR_RESYNC 4 mirrors 000726 1782242304.219568 punch from 0x0087e4 to 0x00d408 (0x4c25 bytes) 000727 1782242304.251232 mapwrite 0x01e5f2 thru 0x027b15 (0x9524 bytes) 000728 1782242304.290655 mapwrite 0x0356c2 thru 0x03cd51 (0x7690 bytes) 000729 1782242304.315483 fallocate 0x02294a thru 0x023a4c (0x1103 bytes) 000731 1782242304.323760 mapread 0x0101d6 thru 0x01c470 (0xc29b bytes) 000732 1782242304.335522 write 0x036870 thru 0x03d7d0 (0x6f61 bytes) 000733 1782242304.341626 MIRROR_RESYNC 4 mirrors 000734 1782242306.015386 punch from 0x000b67 to 0x00dc87 (0xd121 bytes) 000735 1782242306.081100 trunc from 0x040000 to 0x0052e0 (0x3ad1f bytes) 000736 1782242306.119416 trunc from 0x0028c0 to 0x0052e0 (0x2a21 bytes) 000737 1782242306.134525 write 0x00a47d thru 0x00ac64 (0x7e8 bytes) 000738 1782242306.138522 mapwrite 0x022812 thru 0x026ff2 (0x47e1 bytes) 000739 1782242306.282804 fallocate 0x0064b4 thru 0x015746 (0xf293 bytes) 000740 1782242306.302822 mapread 0x0186d8 thru 0x025c22 (0xd54b bytes) 000741 1782242306.335978 mapwrite 0x02088b thru 0x0269e0 (0x6156 bytes) 000742 1782242306.364462 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x26ff3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26ff3): Mirror 2: 0x3af990e5 (stale) Mirror 3: 0x3af990e5 (stale) Mirror 4: 0x3af990e5 (stale) CRC-32 checksum value for chunk [0, 0x26ff3): Mirror 1: 0x19d8a6a7 Mirror 5: 0x19d8a6a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26ff3: skipped 000743 1782242307.503815 trunc from 0x026ff3 to 0x016e82 (0x10170 bytes) 000745 1782242307.593436 write 0x03a095 thru 0x03cf43 (0x2eaf bytes) 000746 1782242307.596073 trunc from 0x03cf44 to 0x01128f (0x2bcb4 bytes) 000747 1782242307.628861 fallocate 0x021688 thru 0x02310d (0x1a86 bytes) 000748 1782242307.637519 mapread 0x011b0f thru 0x01f601 (0xdaf3 bytes) 000749 1782242307.689880 mapread 0x00fbbc thru 0x015170 (0x55b5 bytes) 000750 1782242307.693766 mapwrite 0x03db37 thru 0x03ffff (0x24c9 bytes) 000751 1782242307.744645 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9c028215 (stale) Mirror 3: 0x9c028215 (stale) Mirror 5: 0x912cbebf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9c028215 (stale) Mirror 3: 0x9c028215 (stale) Mirror 5: 0x912cbebf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000752 1782242308.138803 trunc from 0x039daa to 0x03ffff (0x6256 bytes) 000753 1782242308.146423 mapread 0x00b0c4 thru 0x019861 (0xe79e bytes) 000754 1782242308.178166 fallocate 0x006679 thru 0x009227 (0x2baf bytes) 000757 1782242308.186192 punch from 0x033a7b to 0x03460c (0xb92 bytes) 000758 1782242308.206788 trunc from 0x039079 to 0x0391de (0x166 bytes) 000759 1782242308.229789 read 0x02ff06 thru 0x03c0aa (0xc1a5 bytes) 000760 1782242308.242628 punch from 0x039463 to 0x03c2de (0x2e7c bytes) 000762 1782242308.251628 read 0x01fbe4 thru 0x022220 (0x263d bytes) 000764 1782242308.276667 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9c028215 (stale) Mirror 3: 0x9c028215 (stale) Mirror 5: 0x912cbebf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe5ddee50 Mirror 6: 0xe5ddee50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000765 1782242309.427493 trunc from 0x040000 to 0x02db14 (0x124eb bytes) 000766 1782242309.540662 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2db15): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2db15): Mirror 2: 0x48fdab30 (stale) Mirror 3: 0x48fdab30 (stale) Mirror 5: 0xbdc12f4c (stale) Mirror 6: 0xc083e884 (stale) CRC-32 checksum value for chunk [0, 0x2db15): Mirror 1: 0xc083e884 Mirror 7: 0xc083e884 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2db15: skipped 000767 1782242310.837734 fallocate 0x0241a7 thru 0x02a8d7 (0x6731 bytes) 000768 1782242310.886788 mapread 0x028d00 thru 0x02db14 (0x4e15 bytes) 000769 1782242310.912284 punch from 0x02153e to 0x024d18 (0x37db bytes) 000770 1782242310.920018 write 0x0267bc thru 0x033e79 (0xd6be bytes) 000771 1782242310.928803 write 0x031119 thru 0x039bef (0x8ad7 bytes) 000773 1782242310.933769 read 0x00cd11 thru 0x00d26a (0x55a bytes) 000774 1782242310.941980 write 0x02aefa thru 0x02f326 (0x442d bytes) 000775 1782242310.943929 trunc from 0x0168d9 to 0x018ce7 (0x240f bytes) 000776 1782242310.953969 trunc from 0x03026e to 0x0374d9 (0x726c bytes) 000777 1782242310.981478 punch from 0x01347e to 0x01d56a (0xa0ed bytes) 000778 1782242310.990456 read 0x0265ea thru 0x0316c6 (0xb0dd bytes) 000779 1782242311.008706 mapwrite 0x03a2b8 thru 0x03b4df (0x1228 bytes) 000780 1782242311.084180 trunc from 0x02df28 to 0x0369f1 (0x8aca bytes) 000781 1782242311.097320 mapread 0x00fe0b thru 0x018f9d (0x9193 bytes) 000782 1782242311.098894 punch from 0x037374 to 0x03b4df (0x416c bytes) 000783 1782242311.117507 mapread 0x0092d4 thru 0x00a3b8 (0x10e5 bytes) 000784 1782242311.137814 MIRROR_RESYNC 6 mirrors 000785 1782242313.156371 punch from 0x007109 to 0x0115ba (0xa4b2 bytes) 000786 1782242313.211593 fallocate 0x02a710 thru 0x036de3 (0xc6d4 bytes) 000787 1782242313.221707 fallocate 0x03fcc7 thru 0x03ffff (0x339 bytes) 000788 1782242313.242998 mapwrite 0x03949e thru 0x03ffff (0x6b62 bytes) 000789 1782242313.327265 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x641270a7 (stale) Mirror 3: 0x641270a7 (stale) Mirror 5: 0x641270a7 (stale) Mirror 7: 0x641270a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x641270a7 (stale) Mirror 3: 0x641270a7 (stale) Mirror 5: 0x641270a7 (stale) Mirror 7: 0x641270a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000790 1782242313.876120 mapread 0x0338c8 thru 0x03f666 (0xbd9f bytes) 000791 1782242313.944355 fallocate 0x038887 thru 0x03ffff (0x7779 bytes) 000792 1782242313.950987 mapwrite 0x008e23 thru 0x016662 (0xd840 bytes) 000793 1782242314.011275 punch from 0x024b77 to 0x02c70a (0x7b94 bytes) 000794 1782242314.031082 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x641270a7 (stale) Mirror 3: 0x641270a7 (stale) Mirror 5: 0x641270a7 (stale) Mirror 7: 0x641270a7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7346838d Mirror 8: 0x7346838d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000795 1782242315.245499 mapwrite 0x0389fd thru 0x03c0f2 (0x36f6 bytes) 000796 1782242315.408658 punch from 0x0341fa to 0x03ffff (0xbe06 bytes) 000797 1782242315.429010 punch from 0x00050e to 0x00a338 (0x9e2b bytes) 000798 1782242315.443837 mapread 0x035aff thru 0x03c023 (0x6525 bytes) 000799 1782242315.469142 MIRROR_RESYNC 6 mirrors 000800 1782242317.485671 write 0x0001ef thru 0x001c38 (0x1a4a bytes) 000801 1782242317.524539 punch from 0x034e76 to 0x03a4c1 (0x564c bytes) 000802 1782242317.535822 write 0x00736c thru 0x015c79 (0xe90e bytes) 000803 1782242317.560409 write 0x023f1c thru 0x02e3f9 (0xa4de bytes) 000804 1782242317.599094 MIRROR_RESYNC 6 mirrors 000805 1782242319.252315 fallocate 0x00efad thru 0x01a885 (0xb8d9 bytes) 000806 1782242319.316603 trunc from 0x018b54 to 0x01a132 (0x15df bytes) 000808 1782242319.325329 trunc from 0x040000 to 0x00fa73 (0x3058c bytes) 000809 1782242319.359034 mapread 0x0007c0 thru 0x005256 (0x4a97 bytes) 000810 1782242319.402792 trunc from 0x00fa74 to 0x00b151 (0x4922 bytes) 000811 1782242319.427289 MIRROR_RESYNC 6 mirrors 000812 1782242320.732986 write 0x027d2e thru 0x03726a (0xf53d bytes) 000813 1782242320.765315 punch from 0x0239a4 to 0x030a84 (0xd0e1 bytes) 000814 1782242320.785802 mapread 0x000f19 thru 0x005b89 (0x4c71 bytes) 000815 1782242320.807493 trunc from 0x02fa0f to 0x03726a (0x785c bytes) 000816 1782242320.812955 mapwrite 0x0278cf thru 0x0339ab (0xc0dd bytes) 000817 1782242320.857421 read 0x01dc35 thru 0x0255e0 (0x79ac bytes) 000818 1782242320.863630 read 0x02e3c9 thru 0x0334b7 (0x50ef bytes) 000819 1782242320.865122 trunc from 0x03726b to 0x01dc9d (0x195cd bytes) 000820 1782242320.902529 trunc from 0x01dc9e to 0x03cc19 (0x1ef7c bytes) 000821 1782242320.931518 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3cc1a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc1a): Mirror 2: 0xe55069cc (stale) Mirror 3: 0xe55069cc (stale) Mirror 5: 0xe55069cc (stale) Mirror 7: 0xe55069cc (stale) Mirror 8: 0xe55069cc (stale) CRC-32 checksum value for chunk [0, 0x3cc1a): Mirror 1: 0xe55069cc Mirror 9: 0xe55069cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc1a: skipped 000822 1782242322.072580 read 0x013161 thru 0x0225b1 (0xf451 bytes) 000823 1782242322.089615 read 0x01698b thru 0x018491 (0x1b07 bytes) 000824 1782242322.096159 mapread 0x033d29 thru 0x03cc19 (0x8ef1 bytes) 000825 1782242322.128456 write 0x015292 thru 0x022fc6 (0xdd35 bytes) 000826 1782242322.182943 mapread 0x0138bc thru 0x01cfa0 (0x96e5 bytes) 000827 1782242322.194824 write 0x0006af thru 0x00d25d (0xcbaf bytes) 000828 1782242322.207535 mapwrite 0x039ffc thru 0x03e2fe (0x4303 bytes) 000829 1782242322.318245 mapread 0x02b803 thru 0x033fe3 (0x87e1 bytes) 000830 1782242322.388323 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e2ff): Mirror 2: 0x6e1faa04 (stale) Mirror 3: 0x6e1faa04 (stale) Mirror 5: 0x6e1faa04 (stale) Mirror 7: 0x6e1faa04 (stale) Mirror 8: 0x6e1faa04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e2ff): Mirror 2: 0x6e1faa04 (stale) Mirror 3: 0x6e1faa04 (stale) Mirror 5: 0x6e1faa04 (stale) Mirror 7: 0x6e1faa04 (stale) Mirror 8: 0x6e1faa04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e2ff: skipped 000831 1782242322.914559 fallocate 0x023990 thru 0x027a95 (0x4106 bytes) 000832 1782242322.920983 trunc from 0x03e2ff to 0x02f505 (0xedf9 bytes) 000833 1782242322.946691 fallocate 0x0140f2 thru 0x0152ce (0x11dd bytes) 000834 1782242322.952891 fallocate 0x00b41e thru 0x01a2a0 (0xee83 bytes) 000835 1782242322.959659 write 0x00c5dd thru 0x012ad8 (0x64fc bytes) 000836 1782242322.972003 trunc from 0x02f506 to 0x01e911 (0x10bf4 bytes) 000837 1782242322.997811 mapread 0x0128e0 thru 0x0151c1 (0x28e2 bytes) 000838 1782242323.049940 write 0x03c8af thru 0x03d916 (0x1068 bytes) 000839 1782242323.057774 MIRROR_RESYNC 6 mirrors 000840 1782242324.722700 mapwrite 0x01a415 thru 0x01cdde (0x29ca bytes) 000841 1782242324.812921 write 0x006ea5 thru 0x00832b (0x1487 bytes) 000843 1782242324.824012 fallocate 0x03b4dc thru 0x03ffff (0x4b24 bytes) 000844 1782242324.829887 trunc from 0x034e1f to 0x03d916 (0x8af8 bytes) 000845 1782242324.835339 mapread 0x035655 thru 0x03d916 (0x82c2 bytes) 000846 1782242324.848774 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d917): Mirror 3: 0x159aebfd (stale) Mirror 5: 0x159aebfd (stale) Mirror 7: 0x159aebfd (stale) Mirror 8: 0x159aebfd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d917): Mirror 3: 0x159aebfd (stale) Mirror 5: 0x159aebfd (stale) Mirror 7: 0x159aebfd (stale) Mirror 8: 0x159aebfd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d917: skipped 000847 1782242325.533113 write 0x0167fe thru 0x018a07 (0x220a bytes) 000848 1782242325.556683 trunc from 0x03d917 to 0x0241e9 (0x1972d bytes) 000849 1782242325.615540 trunc from 0x0241ea to 0x02947c (0x5293 bytes) 000850 1782242325.638157 mapwrite 0x014c38 thru 0x01f3a7 (0xa770 bytes) 000851 1782242325.719712 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2947d): Mirror 3: 0x615b5614 (stale) Mirror 7: 0x615b5614 (stale) Mirror 8: 0x615b5614 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2947d): Mirror 3: 0x615b5614 (stale) Mirror 7: 0x615b5614 (stale) Mirror 8: 0x615b5614 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2947d: skipped 000852 1782242326.223896 trunc from 0x006cc6 to 0x0105ed (0x9928 bytes) 000853 1782242326.229243 mapread 0x01ad77 thru 0x01d113 (0x239d bytes) 000854 1782242326.251027 mapwrite 0x015a25 thru 0x023c6a (0xe246 bytes) 000856 1782242326.279993 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2947d): Mirror 7: 0x615b5614 (stale) Mirror 8: 0x615b5614 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2947d): Mirror 7: 0x615b5614 (stale) Mirror 8: 0x615b5614 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2947d: skipped 000857 1782242326.745355 write 0x039abb thru 0x03ffff (0x6545 bytes) 000858 1782242326.752320 write 0x03c884 thru 0x03ffff (0x377c bytes) 000859 1782242326.760788 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x4c4ec56a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x4c4ec56a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000860 1782242327.183024 trunc from 0x03cf40 to 0x03ffff (0x30c0 bytes) 000861 1782242327.192921 trunc from 0x040000 to 0x008b84 (0x3747b bytes) 000862 1782242327.236974 fallocate 0x00247c thru 0x00d8ca (0xb44f bytes) 000863 1782242327.249666 trunc from 0x008b85 to 0x00040c (0x8778 bytes) 000864 1782242327.288482 mapwrite 0x00d0fd thru 0x016cc5 (0x9bc9 bytes) 000865 1782242327.459458 read 0x009930 thru 0x00c566 (0x2c37 bytes) 000866 1782242327.460297 trunc from 0x016cc6 to 0x02d362 (0x1669d bytes) 000867 1782242327.489462 trunc from 0x0152cb to 0x022b8d (0xd8c3 bytes) 000869 1782242327.507482 read 0x01abc5 thru 0x02a8fc (0xfd38 bytes) 000870 1782242327.551459 fallocate 0x027911 thru 0x02fd5b (0x844b bytes) 000871 1782242327.566349 punch from 0x00816f to 0x0132e0 (0xb172 bytes) 000872 1782242327.581407 mapread 0x01f1c4 thru 0x02c8c9 (0xd706 bytes) 000873 1782242327.584267 read 0x006395 thru 0x0155bd (0xf229 bytes) 000874 1782242327.600598 trunc from 0x00b8e4 to 0x00d1ea (0x1907 bytes) 000875 1782242327.614395 write 0x0013cb thru 0x003383 (0x1fb9 bytes) 000877 1782242327.618974 fallocate 0x001575 thru 0x00b9da (0xa466 bytes) 000878 1782242327.628365 punch from 0x010fab to 0x01335f (0x23b5 bytes) 000879 1782242327.636960 trunc from 0x007e31 to 0x00ae77 (0x3047 bytes) 000880 1782242327.646351 punch from 0x029ef8 to 0x02d362 (0x346b bytes) 000882 1782242327.653314 mapread 0x027ad7 thru 0x02c324 (0x484e bytes) 000883 1782242327.672934 fallocate 0x02a5fa thru 0x02ed76 (0x477d bytes) 000884 1782242327.679348 trunc from 0x02ed77 to 0x0144bf (0x1a8b7 bytes) 000885 1782242327.702001 read 0x00eb0a thru 0x0144bf (0x59b6 bytes) 000886 1782242327.712667 mapread 0x010630 thru 0x0144bf (0x3e90 bytes) 000887 1782242327.713747 trunc from 0x01059a to 0x0144bf (0x3f26 bytes) 000888 1782242327.720667 trunc from 0x011a08 to 0x0144bf (0x2ab8 bytes) 000889 1782242327.726871 write 0x0296fb thru 0x035349 (0xbc4f bytes) 000891 1782242327.734657 mapread 0x02f69c thru 0x02fa4f (0x3b4 bytes) 000892 1782242327.736458 mapwrite 0x021ae0 thru 0x02465f (0x2b80 bytes) 000893 1782242327.768802 read 0x0024ad thru 0x00d0bd (0xac11 bytes) 000894 1782242327.769850 trunc from 0x03534a to 0x0097f9 (0x2bb50 bytes) 000896 1782242327.791341 MIRROR_RESYNC 2 mirrors 000897 1782242328.397693 MIRROR_RESYNC 2 mirrors 000898 1782242328.860246 read 0x006e6e thru 0x0097f9 (0x298c bytes) 000899 1782242328.868692 punch from 0x000134 to 0x0097f9 (0x96c6 bytes) 000900 1782242328.919446 mapwrite 0x01eede thru 0x02ab4f (0xbc72 bytes) 000901 1782242329.017472 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) 000903 1782242329.236797 mapread 0x01643e thru 0x01e912 (0x84d5 bytes) 000904 1782242329.276000 write 0x0121a3 thru 0x020b43 (0xe9a1 bytes) 000905 1782242329.296094 trunc from 0x02ab50 to 0x0102a5 (0x1a8aa bytes) 000906 1782242329.328500 trunc from 0x008026 to 0x0102a5 (0x8280 bytes) 000907 1782242329.336613 punch from 0x007090 to 0x0080dc (0x104d bytes) 000909 1782242329.355635 mapwrite 0x016125 thru 0x01b560 (0x543c bytes) 000910 1782242329.430364 fallocate 0x00596e thru 0x0096ea (0x3d7d bytes) 000911 1782242329.435948 trunc from 0x004778 to 0x00787b (0x3104 bytes) 000912 1782242329.441196 fallocate 0x03a6a9 thru 0x03beb3 (0x180b bytes) 000914 1782242329.446626 trunc from 0x0190e1 to 0x01b560 (0x2480 bytes) 000915 1782242329.451951 read 0x00f49e thru 0x013f07 (0x4a6a bytes) 000916 1782242329.452966 mapread 0x003098 thru 0x004b10 (0x1a79 bytes) 000917 1782242329.464707 read 0x012007 thru 0x01b560 (0x955a bytes) 000919 1782242329.481897 trunc from 0x01b561 to 0x00316f (0x183f1 bytes) 000920 1782242329.526966 fallocate 0x030c56 thru 0x03db86 (0xcf31 bytes) 000921 1782242329.532726 fallocate 0x0089f1 thru 0x017917 (0xef27 bytes) 000922 1782242329.540212 trunc from 0x002028 to 0x00316f (0x1148 bytes) 000923 1782242329.546851 mapread 0x0017d5 thru 0x00316f (0x199b bytes) 000924 1782242329.561276 write 0x02f6d7 thru 0x03e3ee (0xed18 bytes) 000925 1782242329.563359 punch from 0x025ffd to 0x02e60a (0x860e bytes) 000926 1782242329.568113 trunc from 0x0398e7 to 0x03e3ee (0x4b08 bytes) 000927 1782242329.588570 read 0x005c37 thru 0x012de5 (0xd1af bytes) 000929 1782242329.614000 fallocate 0x00e756 thru 0x01d18b (0xea36 bytes) 000931 1782242329.620822 read 0x0326c3 thru 0x03374a (0x1088 bytes) 000932 1782242329.621741 trunc from 0x01469b to 0x01f8e1 (0xb247 bytes) 000933 1782242329.626908 read 0x012000 thru 0x01b706 (0x9707 bytes) 000934 1782242329.637494 mapwrite 0x02eff3 thru 0x037758 (0x8766 bytes) 000935 1782242329.658254 read 0x024a88 thru 0x029b9a (0x5113 bytes) 000937 1782242329.665467 MIRROR_EXTEND to 2 mirrors 000939 1782242330.398447 trunc from 0x01d7a1 to 0x02af92 (0xd7f2 bytes) 000940 1782242330.436834 fallocate 0x01f7c6 thru 0x0253d0 (0x5c0b bytes) 000941 1782242330.442850 write 0x035dd3 thru 0x039dd3 (0x4001 bytes) 000942 1782242330.456394 trunc from 0x0088d2 to 0x012bbb (0xa2ea bytes) 000943 1782242330.461957 write 0x02577b thru 0x0332c0 (0xdb46 bytes) 000944 1782242330.473348 punch from 0x02a2fb to 0x037297 (0xcf9d bytes) 000945 1782242330.486835 read 0x010a18 thru 0x01dc61 (0xd24a bytes) 000946 1782242330.506015 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) 000948 1782242330.646438 mapread 0x0246eb thru 0x0346d2 (0xffe8 bytes) 000950 1782242330.685916 read 0x01c245 thru 0x029d6c (0xdb28 bytes) 000951 1782242330.698848 fallocate 0x0133d3 thru 0x022e57 (0xfa85 bytes) 000952 1782242330.708629 trunc from 0x03e3ef to 0x0239f9 (0x1a9f5 bytes) 000953 1782242330.747547 trunc from 0x01ab7e to 0x01d699 (0x2b1c bytes) 000954 1782242330.762586 read 0x0097d8 thru 0x016359 (0xcb82 bytes) 000955 1782242330.775984 write 0x012ea9 thru 0x0137cc (0x924 bytes) 000956 1782242330.777344 punch from 0x0230e8 to 0x0239f9 (0x912 bytes) 000957 1782242330.785914 mapread 0x000d20 thru 0x00f8f8 (0xebd9 bytes) 000958 1782242330.788545 trunc from 0x0239fa to 0x023edb (0x4e2 bytes) 000959 1782242330.819564 write 0x02887c thru 0x028c9d (0x422 bytes) 000960 1782242330.824782 trunc from 0x02383f to 0x025201 (0x19c3 bytes) 000961 1782242330.834137 write 0x0007e4 thru 0x00dd2e (0xd54b bytes) 000964 1782242330.843010 read 0x01f139 thru 0x023748 (0x4610 bytes) 000965 1782242330.856311 punch from 0x004bfa to 0x00bf53 (0x735a bytes) 000966 1782242330.890709 fallocate 0x003490 thru 0x0122b6 (0xee27 bytes) 000968 1782242330.911905 fallocate 0x015055 thru 0x01f7cc (0xa778 bytes) 000969 1782242330.938570 trunc from 0x019928 to 0x0228d8 (0x8fb1 bytes) 000972 1782242330.954765 trunc from 0x028c9e to 0x001018 (0x27c85 bytes) 000975 1782242330.979979 mapwrite 0x016127 thru 0x01d67a (0x7554 bytes) 000976 1782242331.102350 punch from 0x01377d to 0x01d67a (0x9efe bytes) 000977 1782242331.111348 trunc from 0x012de0 to 0x013fe0 (0x1201 bytes) 000979 1782242331.121618 mapread 0x00de66 thru 0x01878a (0xa925 bytes) 000981 1782242331.215272 write 0x00c17b thru 0x01008c (0x3f12 bytes) 000982 1782242331.222813 punch from 0x0039c0 to 0x00807b (0x46bc bytes) 000984 1782242331.243809 MIRROR_EXTEND to 2 mirrors 000985 1782242331.894353 punch from 0x009d6b to 0x019489 (0xf71f bytes) 000986 1782242331.933326 fallocate 0x02899a thru 0x02fe77 (0x74de bytes) 000987 1782242331.938619 trunc from 0x01d67b to 0x03ecc8 (0x2164e bytes) 000988 1782242331.984341 punch from 0x0302ee to 0x03c538 (0xc24b bytes) 000989 1782242331.989715 mapwrite 0x00e323 thru 0x01b393 (0xd071 bytes) 000990 1782242332.042681 read 0x02dc78 thru 0x02ede8 (0x1171 bytes) 000992 1782242332.056755 mapwrite 0x0067e5 thru 0x010a28 (0xa244 bytes) 000996 1782242332.102977 read 0x012e00 thru 0x01e71a (0xb91b bytes) 000997 1782242332.104381 punch from 0x0314fd to 0x03907f (0x7b83 bytes) 000998 1782242332.113525 punch from 0x00de44 to 0x011d86 (0x3f43 bytes) 000999 1782242332.118829 trunc from 0x00bc7a to 0x00eea1 (0x3228 bytes) 001000 1782242332.128595 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x197:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (88s) == sanity-flr test 71: check mirror extend parent fid ==== 15:18:57 (1782242337) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1ce:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 15:19:06 (1782242346) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.286545 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100436 s, 10.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 15:19:17 (1782242357) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1096000 .. Extending file size to 8867456 .. failed Extending file size to 4571232 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7301728 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8252480 .. Extending file size to 8740736 .. Extending file size to 8821600 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61127 61128 61130 61131 61132 done 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) 192.168.203.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:20:40 (1782242440) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8326656 .. Extending file size to 7984832 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8500000 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8280864 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9081888 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62553 62554 62555 62557 62559 done 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) 192.168.203.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:22:06 (1782242526) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2160 1269144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25448 3581572 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17760 3589260 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43208 7170832 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2160 1269144 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25448 3581572 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17760 3589260 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43208 7170832 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0898587 s, 23.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227323 s, 9.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x19cbfd73 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:22:24 (1782242544) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111801 s, 18.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 15:22:31 (1782242551) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:22:38 (1782242558) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0761412 s, 27.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:22:44 (1782242564) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.098304 s, 21.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:22:53 (1782242573) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117211 s, 17.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:23:25 (1782242605) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg340-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642843 s, 16.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x241:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x240:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x241:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x240:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23e:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:24:14 (1782242654) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg340-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121395 s, 17.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:25:08 (1782242708) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg340-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg340-server: Pool lustre.test_204f created Adding targets to pool oleg340-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg340-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130871 s, 16.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg340-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg340-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg340-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:26:23 (1782242783) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:26:31 (1782242791) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:26:38 (1782242798) oleg340-server: Pool lustre.test_206 created Adding targets to pool oleg340-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg340-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg340-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg340-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg340-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:27:08 (1782242828) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945973 s, 11.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237587 s, 44.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4830b2b6 Mirror 3: 0x4830b2b6 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:27:19 (1782242839) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:27:22 (1782242842) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:27:25 (1782242845) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0776957 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2240000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xba0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xba0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xba0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2250000 Read this (4 bytes): BBBB 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:28:01 (1782242881) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0193732 s, 423 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:28:08 (1782242888) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398442 s, 26.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611264 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-23 15:28:09.000000000 -0400 Modify: 2026-06-23 15:28:09.000000000 -0400 Change: 2026-06-23 15:28:11.000000000 -0400 Birth: 2026-06-23 15:28:10.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:28:15 (1782242895) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232611 s, 45.1 MB/s oleg340-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg340-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg340-server: O/280000401/d26/13a: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:28:36 (1782242916) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.217519 s, 188 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0315216 s, 130 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 23 15:28 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x85:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 23 15:28 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:28:43 (1782242923) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:28:48 (1782242928) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1772 sec =========== 15:28:56 (1782242936) === sanity-flr: start cleanup 15:28:58 (1782242938) === === sanity-flr: finish cleanup 15:29:06 (1782242946) ===