-----============= acceptance-small: sanity-flr ============----- Tue Sep 1 09:29:52 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg316-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:30:02 (1788269402) === oleg316-client.virtnet: executing check_config_client /mnt/lustre oleg316-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg316-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff90db83218000.idle_timeout=debug osc.lustre-OST0001-osc-ffff90db83218000.idle_timeout=debug disable quota as required oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:30:21 (1788269421) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8203] == sanity-flr test 0a: lfs mirror create with -N option == 09:30:22 (1788269422) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:30:30 (1788269430) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:30:31 (1788269431) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:30:33 (1788269433) PASS 0d (14s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:30:47 (1788269447) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:30:49 (1788269449) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 09:30:51 (1788269451) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff90db83218000.stats=clear osc.lustre-OST0001-osc-ffff90db83218000.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:30:56 (1788269456) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:31:02 (1788269462) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:31:07 (1788269467) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:31:09 (1788269469) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:31:13 (1788269473) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 09:31:18 (1788269478) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:31:22 (1788269482) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:31:25 (1788269485) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:31:29 (1788269489) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:31:34 (1788269494) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:31:37 (1788269497) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.250308 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043923 s, 23.9 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:31:41 (1788269501) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0664092 s, 63.2 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:31:45 (1788269505) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:31:51 (1788269511) Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90db83218000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90db83218000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90db83218000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90db83218000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (43s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:32:34 (1788269554) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.188069 s, 33.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90db83218000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90db83218000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (24s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:32:58 (1788269578) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90db83218000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90db83218000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90db83218000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90db83218000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (47s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:33:45 (1788269625) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.69263 s, 31.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 1 09:33 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.656s user 0m0.002s sys 0m0.155s Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90db83218000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90db83218000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.489s user 0m0.000s sys 0m0.172s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90db83218000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90db83218000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.538s user 0m0.001s sys 0m0.168s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90db83218000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90db83218000.ost_server_uuid in FULL state after 0 sec PASS 33b (48s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:34:33 (1788269673) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:34:35 (1788269675) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:34:36 (1788269676) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 09:34:38 (1788269678) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 09:34:42 (1788269682) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.276973 s, 45.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:34:49 (1788269689) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256384 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332335 s, 31.6 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:34:53 (1788269693) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:34:57 (1788269697) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000746846 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000612962 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000708404 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000736596 s, 1.4 MB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 09:35:03 (1788269703) create test files with size 15743194 .. 445bf5c15d6917e006ef391e85d4bc76 - 0f5010523b2dfac00cfacf9441e2be97 - c369e540b09900e7c95c05bfd43c20f6 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre PASS 37 (10s) == sanity-flr test 38: resync ============================ 09:35:13 (1788269713) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:35:20 (1788269720) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:35:23 (1788269723) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429514 s, 24.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191285 s, 54.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 09:35:27 (1788269727) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0423843 s, 49.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.130421 s, 32.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0644553 s, 32.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 09:35:35 (1788269735) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 09:35:37 (1788269737) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:35:38 (1788269738) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176421 s, 59.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 09:35:43 (1788269743) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.125429 s, 25.1 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:35:47 (1788269747) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:35:51 (1788269751) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.131246 s, 79.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:36:03 (1788269763) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff90db82b07000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.204396 s, 51.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff90db84e95000.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:36:10 (1788269770) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (21s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:36:31 (1788269791) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (22s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:36:53 (1788269813) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 09:36:55 (1788269815) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.134353 s, 23.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0914827 s, 34.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd798fd0b (nosync,stale) Mirror 2: 0xd798fd0b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 811c4e497cf0f9a72b8f90e29b33988b - mirror 1 checksum: 415984fe14f6cb37655237a31543b3dd - mirror 2 checksum: 811c4e497cf0f9a72b8f90e29b33988b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-01T09:36:57-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-01T09:36:57-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 811c4e497cf0f9a72b8f90e29b33988b - mirror 1 checksum: 415984fe14f6cb37655237a31543b3dd - mirror 2 checksum: 415984fe14f6cb37655237a31543b3dd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-01T09:36:57-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-01T09:36:57-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 48 (6s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:37:01 (1788269821) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182378 s, 57.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (4s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:37:05 (1788269825) Failing mds1 on oleg316-server Stopping /mnt/lustre-mds1 (opts:) on oleg316-server 09:37:15 (1788269835) shut down facet: mds1 facet_host: oleg316-server facet_failover_host: oleg316-server Failover mds1 to oleg316-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 09:37:32 (1788269852) targets are mounted 09:37:32 (1788269852) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192292 s, 54.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } PASS 50A (37s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:37:42 (1788269862) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0244597 s, 837 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0119133 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:37:47 (1788269867) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0472716 s, 433 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00667327 s, 3.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (6s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:37:53 (1788269873) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00939811 s, 1.7 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:37:57 (1788269877) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0294435 s, 696 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:38:02 (1788269882) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00474196 s, 864 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0385794 s, 106 kB/s PASS 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:38:07 (1788269887) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x12:0x0]: Invalid argument (22) PASS 60b (12s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:38:19 (1788269899) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:38:42 (1788269922) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:39:01 (1788269941) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:39:20 (1788269960) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 09:39:24 (1788269964) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57472 mirror split pid 57473 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57472 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57473 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 09:40:28 (1788270028) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 29 truncating to largest ever: 0x36153 000001 1788270029.619929 trunc from 00000000 to 0x036152 (0x36153 bytes) 000002 1788270029.638578 mapread 0x00ca2a thru 0x00d2bf (0x896 bytes) 000003 1788270029.652287 mapread 0x0289be thru 0x02ddd6 (0x5419 bytes) 000004 1788270029.661657 trunc from 0x0358c3 to 0x036152 (0x890 bytes) 000008 1788270029.675221 MIRROR_EXTEND to 2 mirrors 000009 1788270030.031362 mapwrite 0x01c6d1 thru 0x028cb7 (0xc5e7 bytes) 000010 1788270030.133153 punch from 0x0255f8 to 0x02cff0 (0x79f9 bytes) 000011 1788270030.137815 trunc from 0x033a1e to 0x036152 (0x2735 bytes) 000012 1788270030.146356 MIRROR_RESYNC 2 mirrors 000013 1788270030.541185 write 0x03c3ce thru 0x03ffff (0x3c32 bytes) 000014 1788270030.561439 trunc from 0x03760f to 0x03ffff (0x89f1 bytes) fallocating to largest ever: 0x38e5a 000015 1788270030.571546 fallocate 0x03797d thru 0x038e59 (0x14dd bytes) 000016 1788270030.579948 trunc from 0x011a48 to 0x018bc0 (0x7179 bytes) 000017 1788270030.587382 mapread 0x00efb7 thru 0x00ff60 (0xfaa bytes) 000018 1788270030.597277 trunc from 0x01ba5d to 0x021541 (0x5ae5 bytes) 000019 1788270030.601227 write 0x0250da thru 0x02b207 (0x612e bytes) 000021 1788270030.610357 fallocate 0x01b5d4 thru 0x023ccf (0x86fc bytes) 000022 1788270030.613950 read 0x02eb0b thru 0x03c25e (0xd754 bytes) 000023 1788270030.620490 read 0x01916d thru 0x01d0d0 (0x3f64 bytes) 000024 1788270030.627632 punch from 0x007db0 to 0x01390a (0xbb5b bytes) 000026 1788270030.631428 mapwrite 0x00b8e4 thru 0x015afb (0xa218 bytes) 000027 1788270030.699010 write 0x00069a thru 0x00ca1c (0xc383 bytes) 000028 1788270030.700604 mapwrite 0x0377d3 thru 0x037ca0 (0x4ce bytes) 000029 1788270030.716970 fallocate 0x013f26 thru 0x0149ba (0xa95 bytes) 000030 1788270030.722460 write 0x02e2d9 thru 0x02fccd (0x19f5 bytes) 000031 1788270030.723777 mapread 0x03a778 thru 0x03b2a9 (0xb32 bytes) 000033 1788270030.725358 mapwrite 0x036bf4 thru 0x03f82f (0x8c3c bytes) 000034 1788270030.745405 punch from 0x00f3cd to 0x01428a (0x4ebe bytes) 000035 1788270030.751778 trunc from 0x007995 to 0x00b53d (0x3ba9 bytes) 000036 1788270030.760957 mapread 0x03a6e1 thru 0x03ffff (0x591f bytes) 000037 1788270030.761883 write 0x0396d2 thru 0x03ffff (0x692e bytes) zero_range to largest ever: 0x3bfb4 000038 1788270030.766153 trunc from 0x02ea5d to 0x03bfb3 (0xd557 bytes) 000040 1788270030.782952 write 0x01f98d thru 0x02d702 (0xdd76 bytes) 000041 1788270030.784714 write 0x0167c1 thru 0x01b40a (0x4c4a bytes) 000042 1788270030.785870 trunc from 0x040000 to 0x036d53 (0x92ac bytes) 000043 1788270030.801738 punch from 0x021ca4 to 0x02dbf6 (0xbf53 bytes) 000044 1788270030.819546 write 0x025322 thru 0x0318c7 (0xc5a6 bytes) 000045 1788270030.834987 trunc from 0x036d54 to 0x01a069 (0x1ccea bytes) 000046 1788270030.859028 read 0x001a57 thru 0x001a5d (0x7 bytes) 000047 1788270030.859793 mapread 0x015e67 thru 0x018be8 (0x2d82 bytes) 000048 1788270030.860711 trunc from 0x0116ed to 0x01a069 (0x897d bytes) 000049 1788270030.871818 trunc from 0x01a06a to 0x026f4f (0xcee6 bytes) 000051 1788270030.883667 punch from 0x0124ac to 0x021417 (0xef6c bytes) 000052 1788270030.889634 mapread 0x024234 thru 0x025dd4 (0x1ba1 bytes) 000053 1788270030.898507 write 0x015240 thru 0x01b99c (0x675d bytes) 000054 1788270030.900405 mapread 0x00234b thru 0x009407 (0x70bd bytes) 000055 1788270030.901361 mapread 0x01559c thru 0x023017 (0xda7c bytes) 000056 1788270030.902549 write 0x019e0c thru 0x01bc84 (0x1e79 bytes) 000057 1788270030.904941 read 0x00e140 thru 0x01351f (0x53e0 bytes) 000058 1788270030.906340 mapread 0x02641e thru 0x026f4f (0xb32 bytes) 000059 1788270030.907808 fallocate 0x02a49e thru 0x038b8d (0xe6f0 bytes) fallocating to largest ever: 0x40000 000060 1788270030.911312 fallocate 0x03fd68 thru 0x03ffff (0x298 bytes) 000061 1788270030.914615 write 0x023e0a thru 0x02bc87 (0x7e7e bytes) 000062 1788270030.915974 mapwrite 0x02ed8b thru 0x0330ab (0x4321 bytes) 000063 1788270030.954967 mapread 0x00dd7a thru 0x01cecb (0xf152 bytes) 000064 1788270030.956021 trunc from 0x01c738 to 0x0200e6 (0x39af bytes) 000065 1788270030.961978 trunc from 0x0330ac to 0x03435d (0x12b2 bytes) 000067 1788270030.978010 read 0x03399d thru 0x03435d (0x9c1 bytes) 000068 1788270030.984599 read 0x005a91 thru 0x007fcc (0x253c bytes) 000070 1788270030.985788 read 0x030354 thru 0x03435d (0x400a bytes) 000071 1788270030.988368 read 0x033009 thru 0x03435d (0x1355 bytes) 000072 1788270030.990717 punch from 0x00fa91 to 0x01a6b1 (0xac21 bytes) 000074 1788270030.996240 trunc from 0x03435e to 0x01558c (0x1edd1 bytes) 000075 1788270031.007293 read 0x000396 thru 0x005b40 (0x57ab bytes) 000078 1788270031.009634 punch from 0x013d3f to 0x01558c (0x184e bytes) 000080 1788270031.017024 mapwrite 0x0294aa thru 0x030ab3 (0x760a bytes) 000081 1788270031.066186 trunc from 0x030ab4 to 0x00a35f (0x26754 bytes) 000082 1788270031.082369 mapwrite 0x0227fa thru 0x02c549 (0x9d50 bytes) 000083 1788270031.170112 mapwrite 0x03c4f1 thru 0x03ffff (0x3b0f bytes) 000084 1788270031.213716 fallocate 0x01c82d thru 0x01c8c3 (0x97 bytes) 000086 1788270031.218502 trunc from 0x040000 to 0x00ac3d (0x353c2 bytes) 000087 1788270031.230794 punch from 0x004496 to 0x00a464 (0x5fcf bytes) 000088 1788270031.236807 trunc from 0x00ac3e to 0x03c305 (0x316c8 bytes) 000089 1788270031.249445 write 0x005735 thru 0x007b2c (0x23f8 bytes) 000090 1788270031.270084 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) 000094 1788270031.398450 mapwrite 0x020a1a thru 0x0297e2 (0x8dc9 bytes) 000095 1788270031.436523 trunc from 0x014b85 to 0x018033 (0x34af bytes) 000096 1788270031.444255 fallocate 0x0108bc thru 0x01fcc8 (0xf40d bytes) 000097 1788270031.450891 trunc from 0x03c306 to 0x0004d6 (0x3be2f bytes) 000099 1788270031.475353 mapwrite 0x03d2fd thru 0x03ffff (0x2d03 bytes) 000100 1788270031.514815 mapread 0x01332d thru 0x01e4d9 (0xb1ad bytes) 000101 1788270031.532445 trunc from 0x040000 to 0x008a52 (0x375ad bytes) 000102 1788270031.556340 fallocate 0x02c378 thru 0x02ffec (0x3c75 bytes) 000103 1788270031.566487 trunc from 0x02ffed to 0x003496 (0x2cb56 bytes) 000104 1788270031.584547 trunc from 0x003497 to 0x02600a (0x22b74 bytes) 000105 1788270031.595263 write 0x039700 thru 0x03ffff (0x6900 bytes) 000106 1788270031.597106 mapread 0x02db3f thru 0x03c43c (0xe8fe bytes) 000107 1788270031.608967 write 0x025cce thru 0x02b30d (0x5640 bytes) 000108 1788270031.613361 trunc from 0x015fbc to 0x016e63 (0xea8 bytes) 000110 1788270031.618211 read 0x02c904 thru 0x03757d (0xac7a bytes) 000111 1788270031.618777 mapwrite 0x036af0 thru 0x03ffff (0x9510 bytes) 000113 1788270031.636399 punch from 0x037a7a to 0x03e343 (0x68ca bytes) 000114 1788270031.650250 MIRROR_EXTEND to 2 mirrors 000115 1788270032.157432 trunc from 0x020b23 to 0x027b98 (0x7076 bytes) 000117 1788270032.187157 trunc from 0x040000 to 0x033a5d (0xc5a2 bytes) 000118 1788270032.216065 read 0x028c7a thru 0x02beda (0x3261 bytes) 000119 1788270032.220906 read 0x030008 thru 0x033a5d (0x3a56 bytes) 000120 1788270032.232229 mapwrite 0x0058c9 thru 0x01239c (0xcad4 bytes) 000121 1788270032.251363 trunc from 0x033a5e to 0x03d139 (0x96dc bytes) 000122 1788270032.262152 trunc from 0x03d13a to 0x011340 (0x2bdf9 bytes) 000123 1788270032.273765 mapwrite 0x02a3d4 thru 0x033ab6 (0x96e3 bytes) 000124 1788270032.342065 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) 000125 1788270032.425976 punch from 0x019f74 to 0x0213f9 (0x7486 bytes) 000128 1788270032.430528 fallocate 0x014c07 thru 0x01e5f2 (0x99ec bytes) 000129 1788270032.433766 mapwrite 0x017791 thru 0x01ee9e (0x770e bytes) 000130 1788270032.475788 mapwrite 0x0175a8 thru 0x024c5b (0xd6b4 bytes) 000132 1788270032.499112 fallocate 0x001ee9 thru 0x00bb36 (0x9c4e bytes) 000133 1788270032.507774 write 0x03adc7 thru 0x03ffff (0x5239 bytes) 000134 1788270032.516988 MIRROR_EXTEND to 2 mirrors 000135 1788270032.989030 trunc from 0x00f3d9 to 0x00fae2 (0x70a bytes) 000136 1788270033.025569 punch from 0x025e06 to 0x02d06a (0x7265 bytes) 000137 1788270033.038069 write 0x037a29 thru 0x039009 (0x15e1 bytes) 000138 1788270033.052021 mapread 0x00adb7 thru 0x011804 (0x6a4e bytes) 000140 1788270033.081612 mapread 0x0217bf thru 0x02f5b4 (0xddf6 bytes) 000142 1788270033.112167 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) 000143 1788270033.210166 read 0x00016b thru 0x007f8f (0x7e25 bytes) 000144 1788270033.219228 fallocate 0x032b44 thru 0x032e36 (0x2f3 bytes) 000145 1788270033.234207 MIRROR_EXTEND to 2 mirrors 000146 1788270033.710198 read 0x00b0b0 thru 0x019215 (0xe166 bytes) 000147 1788270033.722623 mapwrite 0x001a11 thru 0x01094a (0xef3a bytes) 000149 1788270033.823686 MIRROR_RESYNC 2 mirrors 000150 1788270034.242392 trunc from 0x040000 to 0x024b66 (0x1b499 bytes) 000151 1788270034.261566 write 0x032b6b thru 0x03e0d6 (0xb56c bytes) 000152 1788270034.263136 trunc from 0x03e0d7 to 0x020821 (0x1d8b5 bytes) 000153 1788270034.275017 fallocate 0x03c38e thru 0x03ffff (0x3c72 bytes) 000154 1788270034.280545 mapwrite 0x02dc17 thru 0x030193 (0x257d bytes) 000155 1788270034.322147 mapread 0x00a18b thru 0x00e6e0 (0x4556 bytes) 000156 1788270034.328810 mapread 0x02aaf4 thru 0x02fe83 (0x5390 bytes) 000157 1788270034.329626 fallocate 0x01fa55 thru 0x0228b0 (0x2e5c bytes) 000158 1788270034.333240 fallocate 0x00b2e6 thru 0x0140d4 (0x8def bytes) 000159 1788270034.336445 read 0x003d25 thru 0x00e258 (0xa534 bytes) 000160 1788270034.336905 punch from 0x02c28c to 0x02f68b (0x3400 bytes) 000161 1788270034.341491 trunc from 0x030194 to 0x030290 (0xfd bytes) 000162 1788270034.362092 read 0x0102f9 thru 0x011fe2 (0x1cea bytes) 000163 1788270034.363159 trunc from 0x01fbc4 to 0x021c7a (0x20b7 bytes) 000164 1788270034.366583 mapread 0x017ffb thru 0x01eaad (0x6ab3 bytes) 000165 1788270034.375217 trunc from 0x030291 to 0x021c70 (0xe620 bytes) 000166 1788270034.387022 mapread 0x0033bc thru 0x00ecec (0xb931 bytes) 000167 1788270034.387998 read 0x00ce53 thru 0x014cb0 (0x7e5e bytes) 000168 1788270034.388501 punch from 0x01edb7 to 0x021c70 (0x2eba bytes) 000169 1788270034.393864 trunc from 0x00e5b5 to 0x0117b1 (0x31fd bytes) 000170 1788270034.398694 mapread 0x01088b thru 0x01d2ab (0xca21 bytes) 000171 1788270034.404305 trunc from 0x00f00c to 0x00fc98 (0xc8d bytes) 000172 1788270034.408805 write 0x01ee91 thru 0x026890 (0x7a00 bytes) 000173 1788270034.410516 trunc from 0x026891 to 0x037061 (0x107d1 bytes) 000174 1788270034.429401 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37062): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37062): Mirror 2: 0x116dce07 (stale) CRC-32 checksum value for chunk [0, 0x37062): Mirror 1: 0x7a48874a Mirror 3: 0x7a48874a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37062: skipped 000175 1788270034.911763 mapwrite 0x01f84c thru 0x02352e (0x3ce3 bytes) 000177 1788270034.987394 MIRROR_RESYNC 3 mirrors 000178 1788270035.492016 trunc from 0x037062 to 0x00c8a2 (0x2a7bf bytes) 000179 1788270035.517260 read 0x0071ed thru 0x00c6fc (0x5510 bytes) 000180 1788270035.522863 read 0x006596 thru 0x00c8a2 (0x630d bytes) 000181 1788270035.528939 trunc from 0x00c8a3 to 0x03fba6 (0x33304 bytes) 000182 1788270035.544503 write 0x02d645 thru 0x03799f (0xa35b bytes) 000183 1788270035.558442 mapread 0x03a480 thru 0x03c60d (0x218e bytes) 000184 1788270035.559591 mapread 0x01f517 thru 0x02ab2b (0xb615 bytes) 000185 1788270035.561338 mapwrite 0x002f77 thru 0x0036db (0x765 bytes) 000186 1788270035.578916 write 0x020982 thru 0x02a995 (0xa014 bytes) 000187 1788270035.580804 fallocate 0x032603 thru 0x03ae4d (0x884b bytes) 000188 1788270035.584996 write 0x030696 thru 0x035d48 (0x56b3 bytes) 000189 1788270035.586655 trunc from 0x014063 to 0x01b110 (0x70ae bytes) 000190 1788270035.594060 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fba7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fba7): Mirror 2: 0x9017eef3 (stale) Mirror 3: 0x9017eef3 (stale) CRC-32 checksum value for chunk [0, 0x3fba7): Mirror 1: 0xbe4bdd8b Mirror 4: 0xbe4bdd8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fba7: skipped 000192 1788270036.128151 mapwrite 0x02e659 thru 0x0367ab (0x8153 bytes) 000193 1788270036.201131 fallocate 0x02d03e thru 0x03106b (0x402e bytes) 000194 1788270036.206585 trunc from 0x035f93 to 0x03fba6 (0x9c14 bytes) 000195 1788270036.210224 mapread 0x032764 thru 0x03293c (0x1d9 bytes) 000196 1788270036.211457 read 0x026ecd thru 0x032433 (0xb567 bytes) 000197 1788270036.215376 mapwrite 0x01f3ed thru 0x02962f (0xa243 bytes) 000198 1788270036.235907 trunc from 0x03a68f to 0x03fba6 (0x5518 bytes) 000199 1788270036.240144 trunc from 0x03fba7 to 0x01792e (0x28278 bytes) 000200 1788270036.271673 trunc from 0x01792f to 0x03b6c1 (0x23d93 bytes) 000201 1788270036.284711 trunc from 0x03b6c2 to 0x0258fa (0x15dc7 bytes) 000202 1788270036.298302 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x258fb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x258fb): Mirror 2: 0xf1dedbe4 (stale) Mirror 3: 0xf1dedbe4 (stale) Mirror 4: 0x4f9e7715 (stale) CRC-32 checksum value for chunk [0, 0x258fb): Mirror 1: 0xda7ea5b6 Mirror 5: 0xda7ea5b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x258fb: skipped 000203 1788270036.843763 trunc from 0x0258fb to 0x016b70 (0xed8a bytes) 000204 1788270036.894899 punch from 0x013bd8 to 0x013f84 (0x3ad bytes) 000205 1788270036.901379 read 0x0151a7 thru 0x016b70 (0x19ca bytes) 000206 1788270036.907171 write 0x01e67e thru 0x029cd5 (0xb658 bytes) 000207 1788270036.916388 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29cd6): Mirror 3: 0x15eec38b (stale) Mirror 4: 0xdc03bccd (stale) Mirror 5: 0x1b70de9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29cd6): Mirror 3: 0x15eec38b (stale) Mirror 4: 0xdc03bccd (stale) Mirror 5: 0x1b70de9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29cd6: skipped 000208 1788270037.163181 mapwrite 0x0061e9 thru 0x012dfd (0xcc15 bytes) 000209 1788270037.212443 read 0x01785b thru 0x01f6c2 (0x7e68 bytes) 000210 1788270037.223906 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29cd6): Mirror 4: 0xdc03bccd (stale) Mirror 5: 0x1b70de9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29cd6): Mirror 4: 0xdc03bccd (stale) Mirror 5: 0x1b70de9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29cd6: skipped 000211 1788270037.442130 mapwrite 0x00d887 thru 0x00f13a (0x18b4 bytes) 000213 1788270037.472260 write 0x005480 thru 0x00aa92 (0x5613 bytes) 000214 1788270037.480868 trunc from 0x029cd6 to 0x024ada (0x51fb bytes) 000215 1788270037.490499 trunc from 0x024adb to 0x02a735 (0x5c5b bytes) 000216 1788270037.505282 fallocate 0x00dd52 thru 0x017e46 (0xa0f5 bytes) 000217 1788270037.509794 trunc from 0x02a736 to 0x029f90 (0x7a5 bytes) 000218 1788270037.522338 punch from 0x014a0b to 0x018e9a (0x4490 bytes) 000219 1788270037.529448 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29f91): Mirror 5: 0xe80a6b90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29f91): Mirror 5: 0xe80a6b90 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29f91: skipped 000220 1788270037.716595 punch from 0x006993 to 0x009327 (0x2995 bytes) 000221 1788270037.721142 read 0x00e832 thru 0x013e15 (0x55e4 bytes) 000222 1788270037.728634 trunc from 0x029f91 to 0x001c9f (0x282f1 bytes) 000224 1788270037.746269 read 0x00112e thru 0x001c9f (0xb72 bytes) 000225 1788270037.751526 fallocate 0x005ab2 thru 0x0085a9 (0x2af8 bytes) 000226 1788270037.755887 write 0x002e13 thru 0x011a0e (0xebfc bytes) 000227 1788270037.769286 trunc from 0x00abbf to 0x011a0e (0x6e50 bytes) 000228 1788270037.782594 trunc from 0x00abf5 to 0x011a0e (0x6e1a bytes) 000229 1788270037.786449 punch from 0x0110e9 to 0x011a0e (0x926 bytes) 000231 1788270037.790434 punch from 0x004f49 to 0x00ec87 (0x9d3f bytes) 000232 1788270037.795440 trunc from 0x00cf6f to 0x011a0e (0x4aa0 bytes) 000233 1788270037.803149 trunc from 0x00f4fc to 0x011a0e (0x2513 bytes) 000234 1788270037.808294 trunc from 0x004635 to 0x00d639 (0x9005 bytes) 000235 1788270037.812803 read 0x00d542 thru 0x011a0e (0x44cd bytes) 000237 1788270037.824383 mapwrite 0x00cb59 thru 0x00cd11 (0x1b9 bytes) 000239 1788270037.835435 trunc from 0x00dd26 to 0x011a0e (0x3ce9 bytes) 000240 1788270037.840596 mapwrite 0x03fe07 thru 0x03ffff (0x1f9 bytes) 000241 1788270037.877489 punch from 0x013967 to 0x01723b (0x38d5 bytes) 000242 1788270037.885594 read 0x0213d6 thru 0x0299d8 (0x8603 bytes) 000243 1788270037.887368 punch from 0x02276e to 0x02a50e (0x7da1 bytes) 000244 1788270037.892446 fallocate 0x0189d3 thru 0x02586b (0xce99 bytes) 000246 1788270037.897598 punch from 0x02d498 to 0x03184c (0x43b5 bytes) 000248 1788270037.901407 write 0x02745a thru 0x035dce (0xe975 bytes) 000249 1788270037.911686 trunc from 0x040000 to 0x031774 (0xe88b bytes) 000250 1788270037.933819 MIRROR_RESYNC 2 mirrors 000251 1788270038.492806 trunc from 0x031775 to 0x008476 (0x292fe bytes) 000252 1788270038.533032 trunc from 0x003fbc to 0x008476 (0x44bb bytes) 000253 1788270038.538116 punch from 0x0010fd to 0x0079d6 (0x68da bytes) 000254 1788270038.546042 trunc from 0x000c03 to 0x008476 (0x7874 bytes) 000256 1788270038.553226 read 0x001fc8 thru 0x008476 (0x64af bytes) 000257 1788270038.571280 trunc from 0x008477 to 0x020b0c (0x18696 bytes) 000258 1788270038.589478 trunc from 0x002472 to 0x0091e8 (0x6d77 bytes) 000259 1788270038.598491 MIRROR_RESYNC 2 mirrors 000260 1788270038.897888 read 0x012197 thru 0x01d618 (0xb482 bytes) 000261 1788270038.907048 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) 000262 1788270038.987745 read 0x008658 thru 0x013cdf (0xb688 bytes) 000263 1788270038.996219 read 0x006be4 thru 0x00e2c3 (0x76e0 bytes) 000264 1788270039.004220 mapread 0x012690 thru 0x014623 (0x1f94 bytes) 000266 1788270039.005635 read 0x008a4f thru 0x00e46c (0x5a1e bytes) 000267 1788270039.006221 trunc from 0x020b0d to 0x00f7ee (0x1131e bytes) 000268 1788270039.047350 write 0x03fe11 thru 0x03ffff (0x1ef bytes) 000269 1788270039.048363 write 0x010866 thru 0x01f3cd (0xeb68 bytes) 000270 1788270039.059766 read 0x034f45 thru 0x03ffff (0xb0bb bytes) 000271 1788270039.060472 read 0x038e72 thru 0x03bfea (0x3179 bytes) 000272 1788270039.060982 trunc from 0x040000 to 0x01b3e8 (0x24c17 bytes) 000273 1788270039.076605 trunc from 0x01b3e9 to 0x01bf0a (0xb22 bytes) 000274 1788270039.089558 punch from 0x014648 to 0x01bf0a (0x78c3 bytes) 000275 1788270039.108805 mapwrite 0x007c77 thru 0x00e487 (0x6811 bytes) 000276 1788270039.137441 trunc from 0x01bf0b to 0x00e31d (0xdbed bytes) 000277 1788270039.153565 trunc from 0x00e31e to 0x023a88 (0x1576b bytes) 000279 1788270039.168486 mapwrite 0x0390ec thru 0x03ffff (0x6f14 bytes) 000280 1788270039.209118 trunc from 0x040000 to 0x02c010 (0x13fef bytes) 000281 1788270039.223394 trunc from 0x028b2f to 0x02c010 (0x34e2 bytes) 000282 1788270039.230290 trunc from 0x00d0db to 0x00f50e (0x2434 bytes) 000283 1788270039.234972 mapread 0x019410 thru 0x02151f (0x8110 bytes) 000285 1788270039.241022 MIRROR_EXTEND to 2 mirrors 000286 1788270039.642298 trunc from 0x00be9a to 0x01b19c (0xf303 bytes) 000287 1788270039.682721 write 0x033458 thru 0x034082 (0xc2b bytes) 000288 1788270039.697971 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34083): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34083): Mirror 2: 0x4aa6c1e6 (stale) CRC-32 checksum value for chunk [0, 0x34083): Mirror 1: 0xd106b23c Mirror 3: 0xd106b23c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34083: skipped 000289 1788270040.214753 read 0x00eca1 thru 0x010dee (0x214e bytes) 000290 1788270040.233294 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34083): Mirror 2: 0x4aa6c1e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34083): Mirror 2: 0x4aa6c1e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34083: skipped 000291 1788270040.418545 write 0x01eee5 thru 0x02260d (0x3729 bytes) 000292 1788270040.440126 write 0x026e64 thru 0x0350a4 (0xe241 bytes) 000293 1788270040.446874 mapread 0x0188a5 thru 0x020982 (0x80de bytes) 000294 1788270040.460022 fallocate 0x034f60 thru 0x03a609 (0x56aa bytes) 000295 1788270040.464885 mapwrite 0x029ad0 thru 0x02e9b3 (0x4ee4 bytes) 000296 1788270040.482578 trunc from 0x0350a5 to 0x03182e (0x3876 bytes) 000298 1788270040.509384 write 0x019624 thru 0x0272e3 (0xdcc0 bytes) 000299 1788270040.511337 punch from 0x002906 to 0x00fcaf (0xd3aa bytes) 000300 1788270040.519381 punch from 0x02073b to 0x022144 (0x1a0a bytes) 000301 1788270040.527536 trunc from 0x02a25c to 0x02c8f9 (0x269e bytes) 000302 1788270040.530761 trunc from 0x03182f to 0x004743 (0x2d0eb bytes) 000305 1788270040.541207 mapwrite 0x009e1a thru 0x00a689 (0x870 bytes) 000306 1788270040.569179 read 0x0008ce thru 0x0008d5 (0x8 bytes) 000307 1788270040.573526 read 0x007062 thru 0x00a689 (0x3628 bytes) 000308 1788270040.574199 trunc from 0x0017a8 to 0x006011 (0x486a bytes) 000309 1788270040.579140 fallocate 0x01a18a thru 0x026f7b (0xcdf2 bytes) 000310 1788270040.586862 mapread 0x003e37 thru 0x0046e1 (0x8ab bytes) 000311 1788270040.595540 write 0x0051e4 thru 0x013384 (0xe1a1 bytes) 000312 1788270040.596886 punch from 0x012e16 to 0x013384 (0x56f bytes) 000313 1788270040.609368 mapwrite 0x00b86d thru 0x00f90a (0x409e bytes) 000314 1788270040.622230 trunc from 0x013385 to 0x01fd5f (0xc9db bytes) 000315 1788270040.631097 punch from 0x01e460 to 0x01fd5f (0x1900 bytes) 000316 1788270040.638522 read 0x003dd6 thru 0x01322e (0xf459 bytes) 000317 1788270040.649130 mapread 0x01c2c1 thru 0x01fd5f (0x3a9f bytes) 000318 1788270040.649966 mapwrite 0x00f92c thru 0x01b22e (0xb903 bytes) 000319 1788270040.668574 MIRROR_RESYNC 2 mirrors 000320 1788270041.097512 mapwrite 0x02c339 thru 0x03421d (0x7ee5 bytes) 000321 1788270041.186421 trunc from 0x02877b to 0x03421d (0xbaa3 bytes) 000322 1788270041.195513 read 0x01517e thru 0x018e6d (0x3cf0 bytes) 000324 1788270041.207051 write 0x035288 thru 0x03ffff (0xad78 bytes) 000325 1788270041.208480 fallocate 0x001ff3 thru 0x00fa69 (0xda77 bytes) 000326 1788270041.217447 trunc from 0x040000 to 0x0393d9 (0x6c26 bytes) 000329 1788270041.253331 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) 000330 1788270041.383009 trunc from 0x0393da to 0x01a205 (0x1f1d4 bytes) 000331 1788270041.409711 read 0x00463d thru 0x00895b (0x431f bytes) 000332 1788270041.416627 mapread 0x01556b thru 0x01a205 (0x4c9b bytes) 000333 1788270041.429387 punch from 0x013717 to 0x01a205 (0x6aef bytes) 000334 1788270041.434359 trunc from 0x00e016 to 0x0171ff (0x91ea bytes) 000335 1788270041.438546 trunc from 0x01a206 to 0x00a3ae (0xfe57 bytes) 000336 1788270041.460642 mapread 0x000fd2 thru 0x00a3ae (0x93dd bytes) 000337 1788270041.470596 punch from 0x008c10 to 0x00a3ae (0x179f bytes) 000338 1788270041.476678 trunc from 0x003472 to 0x00a3ae (0x6f3d bytes) 000339 1788270041.482804 read 0x002b55 thru 0x0076ca (0x4b76 bytes) 000340 1788270041.498812 fallocate 0x032245 thru 0x037c4b (0x5a07 bytes) 000341 1788270041.506657 mapread 0x031df5 thru 0x037c4b (0x5e57 bytes) 000342 1788270041.528351 fallocate 0x0129b9 thru 0x0206f8 (0xdd40 bytes) 000343 1788270041.533285 mapread 0x0228a3 thru 0x0277b0 (0x4f0e bytes) 000344 1788270041.534692 trunc from 0x037c4c to 0x023653 (0x145f8 bytes) 000345 1788270041.552584 read 0x01fc94 thru 0x023653 (0x39c0 bytes) 000346 1788270041.553209 trunc from 0x019d24 to 0x023653 (0x9930 bytes) 000347 1788270041.556786 read 0x0212ae thru 0x02318f (0x1ee2 bytes) 000348 1788270041.568604 fallocate 0x02108f thru 0x02a51b (0x948d bytes) 000349 1788270041.573034 mapread 0x003da6 thru 0x00c8da (0x8b35 bytes) 000350 1788270041.573937 trunc from 0x02a51c to 0x0152ed (0x1522e bytes) 000351 1788270041.595405 mapwrite 0x004a8c thru 0x00faa7 (0xb01c bytes) 000352 1788270041.620689 write 0x02156b thru 0x02bf3a (0xa9d0 bytes) 000354 1788270041.624702 trunc from 0x028096 to 0x02bf3a (0x3ea5 bytes) 000355 1788270041.643131 read 0x00de34 thru 0x013264 (0x5431 bytes) 000356 1788270041.644031 punch from 0x0071c6 to 0x007be9 (0xa24 bytes) 000357 1788270041.653502 trunc from 0x0206bd to 0x0238d9 (0x321d bytes) 000358 1788270041.659105 trunc from 0x00431e to 0x00e59a (0xa27d bytes) 000359 1788270041.668312 mapread 0x00249c thru 0x008c1d (0x6782 bytes) 000360 1788270041.679628 mapwrite 0x00a228 thru 0x015c48 (0xba21 bytes) 000361 1788270041.703608 mapread 0x022178 thru 0x02bf3a (0x9dc3 bytes) 000362 1788270041.705332 trunc from 0x02bf3b to 0x02618f (0x5dab bytes) 000365 1788270041.725707 write 0x022afe thru 0x022c30 (0x133 bytes) 000366 1788270041.726753 punch from 0x02137d to 0x02618f (0x4e13 bytes) 000367 1788270041.738728 trunc from 0x00fae2 to 0x01e305 (0xe824 bytes) 000368 1788270041.744548 mapread 0x018ea0 thru 0x01b967 (0x2ac8 bytes) 000370 1788270041.765273 mapread 0x01b495 thru 0x0227a9 (0x7315 bytes) 000371 1788270041.805168 trunc from 0x026190 to 0x03f92a (0x1979b bytes) 000372 1788270041.825049 mapwrite 0x037d82 thru 0x03ffff (0x827e bytes) 000373 1788270041.863584 write 0x01fa89 thru 0x0231df (0x3757 bytes) 000374 1788270041.867951 mapread 0x015376 thru 0x01e777 (0x9402 bytes) 000377 1788270041.878606 fallocate 0x01d718 thru 0x01f960 (0x2249 bytes) 000378 1788270041.886222 MIRROR_EXTEND to 2 mirrors 000379 1788270042.532745 read 0x01a432 thru 0x01d532 (0x3101 bytes) 000380 1788270042.549194 MIRROR_RESYNC 2 mirrors 000381 1788270042.800026 MIRROR_RESYNC 2 mirrors 000382 1788270042.992148 trunc from 0x040000 to 0x029a32 (0x165cd bytes) 000383 1788270043.028280 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) 000384 1788270043.111810 punch from 0x020675 to 0x02694b (0x62d7 bytes) 000385 1788270043.114703 fallocate 0x00b536 thru 0x018f2c (0xd9f7 bytes) 000386 1788270043.117549 fallocate 0x011605 thru 0x01a0f8 (0x8af4 bytes) 000387 1788270043.121578 write 0x03c8ab thru 0x03ffff (0x3755 bytes) 000388 1788270043.124724 mapwrite 0x03c357 thru 0x03ffff (0x3ca9 bytes) 000389 1788270043.135782 mapread 0x03e019 thru 0x03ee5d (0xe45 bytes) 000390 1788270043.136869 write 0x036529 thru 0x03bd6f (0x5847 bytes) 000391 1788270043.151021 MIRROR_EXTEND to 2 mirrors 000392 1788270043.717357 trunc from 0x040000 to 0x02fb62 (0x1049d bytes) 000393 1788270043.749531 punch from 0x018f78 to 0x01ed82 (0x5e0b bytes) 000394 1788270043.754732 trunc from 0x00b42e to 0x018e68 (0xda3b bytes) 000395 1788270043.757739 fallocate 0x024320 thru 0x029a6e (0x574f bytes) 000396 1788270043.763292 trunc from 0x00bf4c to 0x01a1cf (0xe284 bytes) 000397 1788270043.769331 mapwrite 0x02e83a thru 0x03783d (0x9004 bytes) 000398 1788270043.821744 trunc from 0x00a984 to 0x019bf2 (0xf26f bytes) 000399 1788270043.826222 punch from 0x00bd4e to 0x01594d (0x9c00 bytes) 000400 1788270043.834184 punch from 0x03391d to 0x03783d (0x3f21 bytes) 000401 1788270043.839335 write 0x03a5be thru 0x03dd96 (0x37d9 bytes) 000402 1788270043.841262 trunc from 0x03dd97 to 0x00ac6a (0x3312c bytes) 000403 1788270043.865145 mapread 0x00307b thru 0x00ac6a (0x7bf0 bytes) 000404 1788270043.876076 punch from 0x0023a5 to 0x00ac6a (0x88c6 bytes) 000405 1788270043.883128 fallocate 0x02a692 thru 0x03402b (0x999a bytes) 000406 1788270043.888055 read 0x030ac9 thru 0x03402b (0x3563 bytes) 000408 1788270043.898803 write 0x00e694 thru 0x012e78 (0x47e5 bytes) 000409 1788270043.909600 mapwrite 0x0124b6 thru 0x019dd4 (0x791f bytes) 000411 1788270043.936718 write 0x012ed8 thru 0x015083 (0x21ac bytes) 000412 1788270043.937747 mapread 0x0179a1 thru 0x024fe4 (0xd644 bytes) 000413 1788270043.947391 trunc from 0x02d10d to 0x03402b (0x6f1f bytes) 000414 1788270043.953046 mapread 0x00ad94 thru 0x01aa7a (0xfce7 bytes) 000415 1788270043.968275 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3402c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3402c): Mirror 2: 0x5059926b (stale) CRC-32 checksum value for chunk [0, 0x3402c): Mirror 1: 0x8ef86e5e Mirror 3: 0x8ef86e5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3402c: skipped 000416 1788270044.660985 read 0x02a4d3 thru 0x02acb4 (0x7e2 bytes) 000417 1788270044.675947 MIRROR_RESYNC 3 mirrors 000418 1788270045.131557 mapwrite 0x02a83e thru 0x03a60d (0xfdd0 bytes) 000419 1788270045.202419 trunc from 0x007f2b to 0x016e82 (0xef58 bytes) 000420 1788270045.205958 trunc from 0x01b8d3 to 0x01bd72 (0x4a0 bytes) 000421 1788270045.212530 read 0x00dcad thru 0x012b9f (0x4ef3 bytes) 000422 1788270045.218167 mapwrite 0x03252f thru 0x03ea1b (0xc4ed bytes) 000423 1788270045.253658 punch from 0x01d8d2 to 0x021ca9 (0x43d8 bytes) 000424 1788270045.258564 trunc from 0x03ea1c to 0x0258c1 (0x1915a bytes) 000425 1788270045.271569 trunc from 0x001f7c to 0x006555 (0x45da bytes) 000426 1788270045.274863 trunc from 0x01ec68 to 0x0258c1 (0x6c5a bytes) 000427 1788270045.278436 read 0x0158a3 thru 0x01677b (0xed9 bytes) 000428 1788270045.288862 mapwrite 0x013d53 thru 0x01c8e9 (0x8b97 bytes) 000429 1788270045.309124 read 0x02167f thru 0x0258c1 (0x4243 bytes) 000430 1788270045.309791 mapwrite 0x00c8b8 thru 0x0121db (0x5924 bytes) 000431 1788270045.328130 fallocate 0x02f22c thru 0x03551e (0x62f3 bytes) 000432 1788270045.338496 trunc from 0x0258c2 to 0x03a6e8 (0x14e27 bytes) 000433 1788270045.348469 mapread 0x034d2f thru 0x03a6e8 (0x59ba bytes) 000434 1788270045.358768 write 0x01ff22 thru 0x021714 (0x17f3 bytes) 000435 1788270045.359884 mapwrite 0x011020 thru 0x016a2a (0x5a0b bytes) 000436 1788270045.379298 trunc from 0x03a6e9 to 0x036448 (0x42a0 bytes) 000437 1788270045.391425 trunc from 0x02d4b6 to 0x036448 (0x8f93 bytes) 000438 1788270045.399676 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36449): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36449): Mirror 2: 0x9905ef31 (stale) Mirror 3: 0x9905ef31 (stale) CRC-32 checksum value for chunk [0, 0x36449): Mirror 1: 0x47b1e777 Mirror 4: 0x47b1e777 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36449: skipped 000439 1788270045.974879 trunc from 0x0034be to 0x00817f (0x4cc2 bytes) 000440 1788270046.012768 mapread 0x035e2a thru 0x036448 (0x61f bytes) 000441 1788270046.025039 mapwrite 0x027491 thru 0x03443b (0xcfab bytes) 000443 1788270046.063062 trunc from 0x02ddd8 to 0x032b12 (0x4d3b bytes) 000444 1788270046.070152 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36449): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36449): Mirror 2: 0x9905ef31 (stale) Mirror 3: 0x9905ef31 (stale) Mirror 4: 0x47b1e777 (stale) CRC-32 checksum value for chunk [0, 0x36449): Mirror 1: 0xefe281c2 Mirror 5: 0xefe281c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36449: skipped 000445 1788270046.698770 mapread 0x0224ef thru 0x028f09 (0x6a1b bytes) 000446 1788270046.724647 trunc from 0x019e46 to 0x01a551 (0x70c bytes) 000447 1788270046.756761 mapread 0x024104 thru 0x03338d (0xf28a bytes) 000448 1788270046.772437 mapread 0x015dc4 thru 0x018efe (0x313b bytes) 000449 1788270046.786424 trunc from 0x036449 to 0x005558 (0x30ef0 bytes) 000450 1788270046.844348 fallocate 0x03ec61 thru 0x03ffff (0x139f bytes) 000451 1788270046.849802 write 0x03439b thru 0x03853e (0x41a4 bytes) 000452 1788270046.856120 MIRROR_RESYNC 5 mirrors 000453 1788270047.442267 punch from 0x02482e to 0x02ca39 (0x820c bytes) 000454 1788270047.461011 read 0x0307af thru 0x03853e (0x7d90 bytes) 000455 1788270047.467548 mapwrite 0x026308 thru 0x032bce (0xc8c7 bytes) 000456 1788270047.489955 fallocate 0x026a0c thru 0x035207 (0xe7fc bytes) 000457 1788270047.495061 fallocate 0x012eca thru 0x01bbd4 (0x8d0b bytes) 000458 1788270047.497795 mapread 0x028b0f thru 0x02c1e1 (0x36d3 bytes) 000459 1788270047.502093 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3853f): Mirror 3: 0xc64842e9 (stale) Mirror 4: 0xc64842e9 (stale) Mirror 5: 0xc64842e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3853f): Mirror 3: 0xc64842e9 (stale) Mirror 4: 0xc64842e9 (stale) Mirror 5: 0xc64842e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3853f: skipped 000460 1788270047.707073 mapread 0x027067 thru 0x031eea (0xae84 bytes) 000461 1788270047.719192 MIRROR_RESYNC 4 mirrors 000462 1788270048.289121 read 0x019764 thru 0x01c98f (0x322c bytes) 000463 1788270048.293776 trunc from 0x026633 to 0x02dd3e (0x770c bytes) 000464 1788270048.311081 mapwrite 0x00f036 thru 0x017ace (0x8a99 bytes) 000465 1788270048.336212 mapread 0x019a54 thru 0x026604 (0xcbb1 bytes) 000467 1788270048.348417 read 0x0346b9 thru 0x03479b (0xe3 bytes) 000469 1788270048.355951 trunc from 0x037936 to 0x03853e (0xc09 bytes) 000470 1788270048.361062 read 0x033a77 thru 0x03853e (0x4ac8 bytes) 000471 1788270048.364138 mapread 0x0320af thru 0x03853e (0x6490 bytes) 000472 1788270048.365048 mapread 0x00a10c thru 0x00bf62 (0x1e57 bytes) 000473 1788270048.366503 fallocate 0x01b421 thru 0x029cef (0xe8cf bytes) 000474 1788270048.370510 trunc from 0x000c06 to 0x005a55 (0x4e50 bytes) 000475 1788270048.374057 write 0x00dafe thru 0x01d6a4 (0xfba7 bytes) 000476 1788270048.375509 trunc from 0x03853f to 0x02ac60 (0xd8de bytes) 000477 1788270048.388281 punch from 0x01ad91 to 0x01c244 (0x14b4 bytes) 000478 1788270048.397789 fallocate 0x0391bf thru 0x03da3e (0x4880 bytes) 000479 1788270048.406670 MIRROR_RESYNC 4 mirrors 000481 1788270049.110324 trunc from 0x03da3f to 0x02a75d (0x132e1 bytes) 000482 1788270049.160209 mapwrite 0x004582 thru 0x012611 (0xe090 bytes) 000483 1788270049.204536 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a75e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a75e): Mirror 3: 0xc0fc59bc (stale) Mirror 4: 0xc0fc59bc (stale) Mirror 5: 0xc0fc59bc (stale) CRC-32 checksum value for chunk [0, 0x2a75e): Mirror 1: 0xd766340e Mirror 6: 0xd766340e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a75e: skipped 000484 1788270049.764308 read 0x01f022 thru 0x029f79 (0xaf58 bytes) 000485 1788270049.781064 fallocate 0x02ff2d thru 0x038ce4 (0x8db8 bytes) 000486 1788270049.807587 read 0x0376b6 thru 0x038ce4 (0x162f bytes) 000487 1788270049.817936 punch from 0x02b0ca to 0x02cd61 (0x1c98 bytes) 000488 1788270049.825806 trunc from 0x00e256 to 0x01372e (0x54d9 bytes) 000489 1788270049.829590 fallocate 0x018e69 thru 0x01d2bf (0x4457 bytes) 000490 1788270049.833878 punch from 0x02b234 to 0x0362f0 (0xb0bd bytes) 000491 1788270049.841588 write 0x00b822 thru 0x00f7fa (0x3fd9 bytes) 000492 1788270049.850844 trunc from 0x038ce5 to 0x00581a (0x334ca bytes) 000493 1788270049.883422 mapread 0x004817 thru 0x00581a (0x1004 bytes) 000494 1788270049.889085 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x581b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x581b): Mirror 3: 0xef3da444 (stale) Mirror 4: 0xef3da444 (stale) Mirror 5: 0xef3da444 (stale) Mirror 6: 0x9d5e0bdd (stale) CRC-32 checksum value for chunk [0, 0x581b): Mirror 1: 0x9d5e0bdd Mirror 7: 0x9d5e0bdd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x581b: skipped 000495 1788270050.400671 mapwrite 0x012909 thru 0x0182b0 (0x59a8 bytes) 000496 1788270050.453106 read 0x007d87 thru 0x00b634 (0x38ae bytes) 000497 1788270050.457651 trunc from 0x0182b1 to 0x01ef7b (0x6ccb bytes) 000498 1788270050.467240 read 0x0148be thru 0x01e67e (0x9dc1 bytes) 000499 1788270050.474807 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ef7c): Mirror 4: 0x7f806378 (stale) Mirror 5: 0x7f806378 (stale) Mirror 6: 0x9eacfccf (stale) Mirror 7: 0xb0a62067 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ef7c): Mirror 4: 0x7f806378 (stale) Mirror 5: 0x7f806378 (stale) Mirror 6: 0x9eacfccf (stale) Mirror 7: 0xb0a62067 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ef7c: skipped 000500 1788270050.674307 punch from 0x019f8a to 0x01c48a (0x2501 bytes) 000501 1788270050.676576 read 0x01920c thru 0x019f5f (0xd54 bytes) 000502 1788270050.679147 write 0x014ded thru 0x024705 (0xf919 bytes) 000503 1788270050.691435 write 0x0062fc thru 0x00af46 (0x4c4b bytes) 000504 1788270050.699179 read 0x00244c thru 0x01001b (0xdbd0 bytes) 000505 1788270050.704765 mapwrite 0x01eee0 thru 0x02af15 (0xc036 bytes) 000506 1788270050.742534 punch from 0x01388d to 0x014e7c (0x15f0 bytes) 000507 1788270050.750256 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2af16): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2af16): Mirror 4: 0x88ff2c5a (stale) Mirror 5: 0x88ff2c5a (stale) Mirror 6: 0xa30beb1e (stale) Mirror 7: 0xc30ea88 (stale) CRC-32 checksum value for chunk [0, 0x2af16): Mirror 1: 0xfdb306a4 Mirror 8: 0xfdb306a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2af16: skipped 000508 1788270051.247576 trunc from 0x02af16 to 0x038cce (0xddb9 bytes) 000509 1788270051.298224 fallocate 0x0030f5 thru 0x00706e (0x3f7a bytes) 000510 1788270051.302330 mapwrite 0x00e6bb thru 0x00fb8d (0x14d3 bytes) 000511 1788270051.321207 read 0x010c44 thru 0x0116e2 (0xa9f bytes) 000512 1788270051.325339 trunc from 0x01fdd3 to 0x0223bd (0x25eb bytes) 000513 1788270051.328976 read 0x020f00 thru 0x028ffe (0x80ff bytes) 000514 1788270051.347016 punch from 0x01fd02 to 0x02ea1d (0xed1c bytes) 000515 1788270051.354028 trunc from 0x030783 to 0x031c78 (0x14f6 bytes) 000516 1788270051.358704 write 0x0399b1 thru 0x03ffff (0x664f bytes) 000517 1788270051.360319 fallocate 0x028c9a thru 0x030de8 (0x814f bytes) 000518 1788270051.366586 read 0x0240bc thru 0x028a7b (0x49c0 bytes) 000519 1788270051.376964 mapwrite 0x0221e7 thru 0x0229e4 (0x7fe bytes) 000521 1788270051.393313 MIRROR_RESYNC 6 mirrors 000522 1788270052.463615 trunc from 0x040000 to 0x03250a (0xdaf5 bytes) 000523 1788270052.497233 read 0x02711e thru 0x03250a (0xb3ed bytes) 000524 1788270052.506505 write 0x002f55 thru 0x00a3f4 (0x74a0 bytes) 000525 1788270052.516518 read 0x00a697 thru 0x01529a (0xac04 bytes) 000526 1788270052.522472 mapwrite 0x012896 thru 0x01d97c (0xb0e7 bytes) 000527 1788270052.541024 write 0x034aa2 thru 0x03da90 (0x8fef bytes) 000528 1788270052.542391 fallocate 0x0177ce thru 0x018b3b (0x136e bytes) 000529 1788270052.546058 mapread 0x005bb1 thru 0x006653 (0xaa3 bytes) 000531 1788270052.547525 trunc from 0x03da91 to 0x00d7a8 (0x302e8 bytes) 000532 1788270052.565921 mapwrite 0x00bae4 thru 0x00c639 (0xb56 bytes) 000533 1788270052.576951 write 0x00b096 thru 0x010e93 (0x5dfe bytes) 000534 1788270052.577952 trunc from 0x00109e to 0x00f6d0 (0xe633 bytes) 000535 1788270052.587047 MIRROR_RESYNC 6 mirrors 000536 1788270052.993792 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) 000537 1788270053.171491 read 0x00256b thru 0x00316c (0xc02 bytes) 000539 1788270053.175576 punch from 0x006cb1 to 0x010e93 (0xa1e3 bytes) 000540 1788270053.194110 trunc from 0x010e94 to 0x0067cc (0xa6c7 bytes) 000541 1788270053.214875 fallocate 0x025af6 thru 0x0291bc (0x36c7 bytes) 000542 1788270053.218653 punch from 0x002423 to 0x0067cc (0x43aa bytes) 000543 1788270053.222741 punch from 0x000af2 to 0x00640a (0x5919 bytes) 000544 1788270053.227996 read 0x001157 thru 0x0067cc (0x5676 bytes) 000545 1788270053.244399 trunc from 0x0018bf to 0x0067cc (0x4f0e bytes) 000546 1788270053.248215 trunc from 0x0067cd to 0x031d02 (0x2b536 bytes) 000547 1788270053.263286 write 0x0076fe thru 0x00bbe3 (0x44e6 bytes) 000549 1788270053.273314 fallocate 0x00b107 thru 0x00f19d (0x4097 bytes) 000550 1788270053.278632 write 0x00fde9 thru 0x01659f (0x67b7 bytes) 000551 1788270053.279776 trunc from 0x031d03 to 0x01f6d5 (0x1262d bytes) 000552 1788270053.292581 punch from 0x0107c7 to 0x011336 (0xb70 bytes) 000553 1788270053.307290 write 0x02541c thru 0x0348c4 (0xf4a9 bytes) 000554 1788270053.309187 mapread 0x01abcd thru 0x029039 (0xe46d bytes) 000555 1788270053.327500 trunc from 0x0348c5 to 0x02a85e (0xa066 bytes) 000556 1788270053.340518 MIRROR_RESYNC 5 mirrors 000557 1788270054.042221 fallocate 0x0256fb thru 0x02c5e6 (0x6eec bytes) 000558 1788270054.061598 trunc from 0x00b08a to 0x012bd7 (0x7b4e bytes) 000559 1788270054.064575 read 0x0215fe thru 0x02432f (0x2d32 bytes) 000560 1788270054.067632 mapread 0x00e327 thru 0x017cad (0x9987 bytes) 000561 1788270054.084199 trunc from 0x028dcf to 0x02a85e (0x1a90 bytes) 000562 1788270054.090413 fallocate 0x007f93 thru 0x012f42 (0xafb0 bytes) 000563 1788270054.101391 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a85f): Mirror 5: 0x1e685407 (stale) Mirror 7: 0x1e685407 (stale) Mirror 8: 0x1e685407 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a85f): Mirror 5: 0x1e685407 (stale) Mirror 7: 0x1e685407 (stale) Mirror 8: 0x1e685407 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a85f: skipped 000564 1788270054.307301 read 0x01e806 thru 0x02a85e (0xc059 bytes) 000565 1788270054.313366 mapread 0x01a448 thru 0x021e04 (0x79bd bytes) 000566 1788270054.339540 trunc from 0x02a85f to 0x0300ce (0x5870 bytes) 000568 1788270054.372514 trunc from 0x0300cf to 0x0086e8 (0x279e6 bytes) 000569 1788270054.383258 punch from 0x00343c to 0x0086e8 (0x52ad bytes) 000570 1788270054.388401 trunc from 0x002bab to 0x004648 (0x1a9e bytes) 000571 1788270054.393593 punch from 0x005fee to 0x0086e8 (0x26fb bytes) 000572 1788270054.397226 trunc from 0x005ef7 to 0x0086e8 (0x27f2 bytes) 000573 1788270054.400402 punch from 0x001397 to 0x0086e8 (0x7352 bytes) 000574 1788270054.404329 trunc from 0x0086e9 to 0x02b962 (0x2327a bytes) 000575 1788270054.417598 fallocate 0x00e5f4 thru 0x0123d2 (0x3ddf bytes) 000576 1788270054.421636 punch from 0x02786c to 0x028080 (0x815 bytes) 000577 1788270054.425724 trunc from 0x0207c6 to 0x02743e (0x6c79 bytes) 000578 1788270054.429214 read 0x02014b thru 0x020ef1 (0xda7 bytes) 000580 1788270054.438596 mapread 0x00ae28 thru 0x018c89 (0xde62 bytes) 000581 1788270054.439501 punch from 0x028773 to 0x02b962 (0x31f0 bytes) 000582 1788270054.442986 fallocate 0x03962d thru 0x03c71b (0x30ef bytes) 000583 1788270054.448539 read 0x017483 thru 0x0205f3 (0x9171 bytes) 000584 1788270054.449898 read 0x00d9e8 thru 0x01d4b4 (0xfacd bytes) 000585 1788270054.450635 mapread 0x0294b9 thru 0x0366c9 (0xd211 bytes) 000586 1788270054.462069 write 0x00d4f0 thru 0x017671 (0xa182 bytes) 000587 1788270054.463302 punch from 0x01d484 to 0x02a343 (0xcec0 bytes) 000588 1788270054.469003 fallocate 0x03420f thru 0x037d0d (0x3aff bytes) 000590 1788270054.473309 mapread 0x038575 thru 0x03881c (0x2a8 bytes) 000591 1788270054.474108 trunc from 0x03c71c to 0x0266dd (0x1603e bytes) 000592 1788270054.491765 mapread 0x00e5cd thru 0x00ead6 (0x50a bytes) 000593 1788270054.492861 read 0x00eea7 thru 0x01a5c9 (0xb723 bytes) 000594 1788270054.498521 MIRROR_RESYNC 4 mirrors 000595 1788270055.085213 mapwrite 0x036ef7 thru 0x03b9c9 (0x4ad3 bytes) 000596 1788270055.137661 trunc from 0x03b9ca to 0x03bbe5 (0x21c bytes) 000597 1788270055.150181 write 0x0254e1 thru 0x0327f6 (0xd316 bytes) 000598 1788270055.158047 mapwrite 0x012399 thru 0x015232 (0x2e9a bytes) 000599 1788270055.178944 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bbe6): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bbe6): Mirror 5: 0xcb9bcf6f (stale) Mirror 7: 0xcb9bcf6f (stale) Mirror 8: 0xcb9bcf6f (stale) CRC-32 checksum value for chunk [0, 0x3bbe6): Mirror 1: 0xf0e225dc Mirror 9: 0xf0e225dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bbe6: skipped 000600 1788270055.717805 mapwrite 0x000c36 thru 0x001614 (0x9df bytes) 000601 1788270055.791801 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bbe6): Mirror 5: 0xcb9bcf6f (stale) Mirror 8: 0xcb9bcf6f (stale) Mirror 9: 0xf0e225dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bbe6): Mirror 5: 0xcb9bcf6f (stale) Mirror 8: 0xcb9bcf6f (stale) Mirror 9: 0xf0e225dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bbe6: skipped 000602 1788270055.975176 read 0x018308 thru 0x01d2ea (0x4fe3 bytes) 000603 1788270055.981738 punch from 0x003596 to 0x00a9ae (0x7419 bytes) 000604 1788270055.986732 fallocate 0x01cfc4 thru 0x01ff27 (0x2f64 bytes) 000605 1788270055.990147 trunc from 0x03bbe6 to 0x005dec (0x35df9 bytes) 000606 1788270056.014711 punch from 0x004384 to 0x005dec (0x1a69 bytes) 000607 1788270056.021995 mapread 0x004e02 thru 0x005dec (0xfeb bytes) 000608 1788270056.030405 write 0x03ec36 thru 0x03ffff (0x13ca bytes) 000609 1788270056.031400 mapwrite 0x005d17 thru 0x00f997 (0x9c81 bytes) 000610 1788270056.051649 mapwrite 0x01dc2f thru 0x02acb3 (0xd085 bytes) 000611 1788270056.074320 read 0x00866b thru 0x014ee6 (0xc87c bytes) 000612 1788270056.079925 mapwrite 0x03f37b thru 0x03ffff (0xc85 bytes) 000613 1788270056.088313 trunc from 0x0212f5 to 0x021655 (0x361 bytes) 000614 1788270056.093233 fallocate 0x03cef7 thru 0x03feb1 (0x2fbb bytes) 000615 1788270056.098570 trunc from 0x040000 to 0x02ded8 (0x12127 bytes) 000616 1788270056.107762 mapread 0x01a8c0 thru 0x029bcc (0xf30d bytes) 000618 1788270056.113042 write 0x03a272 thru 0x03ae25 (0xbb4 bytes) 000619 1788270056.113924 mapread 0x0270c8 thru 0x02a24d (0x3186 bytes) 000620 1788270056.114593 mapread 0x00d8b7 thru 0x01375e (0x5ea8 bytes) 000621 1788270056.115349 write 0x03f3c4 thru 0x03ffff (0xc3c bytes) 000622 1788270056.116165 punch from 0x020b34 to 0x02ea95 (0xdf62 bytes) 000623 1788270056.118815 mapwrite 0x037bd8 thru 0x03833f (0x768 bytes) 000625 1788270056.130806 read 0x0181e6 thru 0x0222f3 (0xa10e bytes) 000626 1788270056.139807 MIRROR_RESYNC 4 mirrors 000627 1788270056.901398 write 0x029880 thru 0x0365ae (0xcd2f bytes) 000628 1788270056.950656 mapread 0x0171db thru 0x02269a (0xb4c0 bytes) 000630 1788270056.973166 write 0x00c8d2 thru 0x0192d4 (0xca03 bytes) 000631 1788270056.983276 read 0x00adf1 thru 0x00b78a (0x99a bytes) 000632 1788270056.987238 write 0x0341c6 thru 0x03ffff (0xbe3a bytes) 000633 1788270056.988731 punch from 0x01c1e8 to 0x01fcd7 (0x3af0 bytes) 000634 1788270057.000803 MIRROR_RESYNC 4 mirrors 000635 1788270057.573313 trunc from 0x00e983 to 0x010cd8 (0x2356 bytes) 000636 1788270057.585435 write 0x035890 thru 0x038ee8 (0x3659 bytes) 000638 1788270057.596659 MIRROR_RESYNC 4 mirrors 000639 1788270058.286594 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) 000640 1788270058.501524 trunc from 0x040000 to 0x02343d (0x1cbc2 bytes) 000642 1788270058.529325 punch from 0x00d885 to 0x014462 (0x6bde bytes) 000643 1788270058.532779 fallocate 0x0155b5 thru 0x01cd5b (0x77a7 bytes) 000644 1788270058.535911 trunc from 0x02343e to 0x0218a6 (0x1b97 bytes) 000645 1788270058.562236 mapread 0x001f58 thru 0x005e5c (0x3f05 bytes) 000646 1788270058.588039 MIRROR_RESYNC 3 mirrors 000647 1788270059.114114 mapwrite 0x0098c6 thru 0x00ebf0 (0x532b bytes) 000648 1788270059.181122 read 0x01aee2 thru 0x01da55 (0x2b74 bytes) 000649 1788270059.184667 trunc from 0x013f94 to 0x01906f (0x50dc bytes) 000650 1788270059.187561 mapwrite 0x014250 thru 0x0168f1 (0x26a2 bytes) 000651 1788270059.205165 trunc from 0x01a571 to 0x0218a6 (0x7336 bytes) 000652 1788270059.208923 trunc from 0x0218a7 to 0x02b1be (0x9918 bytes) 000654 1788270059.226949 read 0x00508d thru 0x00651d (0x1491 bytes) 000655 1788270059.230493 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b1bf): Mirror 9: 0x3a5965c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b1bf): Mirror 9: 0x3a5965c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b1bf: skipped 000656 1788270059.366190 mapread 0x028651 thru 0x02b1be (0x2b6e bytes) 000657 1788270059.375991 punch from 0x00af53 to 0x01900f (0xe0bd bytes) 000658 1788270059.379957 write 0x02eebb thru 0x039e5c (0xafa2 bytes) 000659 1788270059.384510 write 0x0280e5 thru 0x02df9f (0x5ebb bytes) 000660 1788270059.388483 trunc from 0x002e51 to 0x00d909 (0xaab9 bytes) 000661 1788270059.394938 write 0x03a5b3 thru 0x03ffff (0x5a4d bytes) 000662 1788270059.396624 punch from 0x023f98 to 0x0317b6 (0xd81f bytes) 000663 1788270059.412188 mapwrite 0x016061 thru 0x022fbc (0xcf5c bytes) 000664 1788270059.453426 punch from 0x0067ef to 0x00b641 (0x4e53 bytes) 000665 1788270059.458648 read 0x021f49 thru 0x02697c (0x4a34 bytes) 000666 1788270059.459694 trunc from 0x035af7 to 0x03ed5d (0x9267 bytes) 000667 1788270059.464208 trunc from 0x040000 to 0x0249d5 (0x1b62a bytes) 000668 1788270059.482683 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x249d6): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x249d6): Mirror 9: 0x54e386b0 (stale) CRC-32 checksum value for chunk [0, 0x249d6): Mirror 1: 0x41e28b96 Mirror 10: 0x41e28b96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x249d6: skipped 000669 1788270059.868575 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x249d6): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x249d6): Mirror 9: 0x54e386b0 (stale) CRC-32 checksum value for chunk [0, 0x249d6): Mirror 1: 0x41e28b96 Mirror 10: 0x41e28b96 Mirror 11: 0x41e28b96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x249d6: skipped 000670 1788270060.311269 fallocate 0x0102a1 thru 0x010e89 (0xbe9 bytes) 000672 1788270060.333475 punch from 0x01745b to 0x0249d5 (0xd57b bytes) 000673 1788270060.336359 write 0x00f10a thru 0x01eb72 (0xfa69 bytes) 000674 1788270060.344989 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x249d6): Mirror 9: 0x54e386b0 (stale) Mirror 11: 0x41e28b96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x249d6): Mirror 9: 0x54e386b0 (stale) Mirror 11: 0x41e28b96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x249d6: skipped 000675 1788270060.499663 write 0x00567c thru 0x014d80 (0xf705 bytes) 000676 1788270060.507786 mapread 0x017e80 thru 0x01c658 (0x47d9 bytes) 000677 1788270060.526004 trunc from 0x01fc64 to 0x0249d5 (0x4d72 bytes) 000678 1788270060.531567 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x249d6): Mirror 11: 0x41e28b96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x249d6): Mirror 11: 0x41e28b96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x249d6: skipped 000679 1788270060.700794 trunc from 0x0249d6 to 0x028780 (0x3dab bytes) 000680 1788270060.729774 write 0x01f234 thru 0x02e50e (0xf2db bytes) 000681 1788270060.734358 trunc from 0x02e50f to 0x02b1ca (0x3344 bytes) 000682 1788270060.745283 MIRROR_RESYNC 2 mirrors 000683 1788270061.050626 write 0x023d2b thru 0x02ae4e (0x7124 bytes) 000684 1788270061.073354 punch from 0x0161bb to 0x01b091 (0x4ed7 bytes) 000685 1788270061.076253 fallocate 0x00b0a2 thru 0x015365 (0xa2c4 bytes) 000688 1788270061.079975 mapread 0x02ac12 thru 0x02b1ca (0x5b9 bytes) 000689 1788270061.084154 trunc from 0x0047a4 to 0x00d1b3 (0x8a10 bytes) 000690 1788270061.090671 fallocate 0x032209 thru 0x039ad5 (0x78cd bytes) 000691 1788270061.094895 trunc from 0x02b1cb to 0x01daf8 (0xd6d2 bytes) 000692 1788270061.109449 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1daf9): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1daf9): Mirror 11: 0xaae1acf2 (stale) CRC-32 checksum value for chunk [0, 0x1daf9): Mirror 1: 0xe682fff3 Mirror 12: 0xe682fff3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1daf9: skipped 000693 1788270061.511221 trunc from 0x01daf9 to 0x022a37 (0x4f3f bytes) 000694 1788270061.538539 MIRROR_RESYNC 3 mirrors 000695 1788270061.954104 mapwrite 0x02263c thru 0x032042 (0xfa07 bytes) 000696 1788270062.001735 fallocate 0x005f84 thru 0x01431c (0xe399 bytes) 000697 1788270062.006680 MIRROR_RESYNC 3 mirrors 000698 1788270062.392671 trunc from 0x02b610 to 0x02c25d (0xc4e bytes) 000699 1788270062.406393 mapwrite 0x023c29 thru 0x02e474 (0xa84c bytes) 000701 1788270062.429025 trunc from 0x032043 to 0x01acf4 (0x1734e bytes) 000702 1788270062.443196 mapread 0x00a5db thru 0x01a3e4 (0xfe0a bytes) 000703 1788270062.454993 read 0x0045ee thru 0x00b316 (0x6d29 bytes) 000704 1788270062.455448 write 0x015771 thru 0x0190aa (0x393a bytes) 000705 1788270062.456309 write 0x0169e7 thru 0x026522 (0xfb3c bytes) 000706 1788270062.457807 write 0x0014dc thru 0x00bc09 (0xa72e bytes) 000707 1788270062.462520 fallocate 0x00706c thru 0x00f8b3 (0x8848 bytes) 000710 1788270062.467107 trunc from 0x026523 to 0x024a9c (0x1a86 bytes) 000711 1788270062.480857 trunc from 0x0170fd to 0x0176d3 (0x5d7 bytes) 000713 1788270062.493187 mapwrite 0x03b3a9 thru 0x03da66 (0x26be bytes) 000714 1788270062.515436 mapwrite 0x00178d thru 0x0042f1 (0x2b65 bytes) 000715 1788270062.524524 write 0x00dbce thru 0x017e76 (0xa2a9 bytes) 000716 1788270062.528104 trunc from 0x03da67 to 0x017903 (0x26163 bytes) 000718 1788270062.538486 trunc from 0x00a486 to 0x00cbe0 (0x275b bytes) 000719 1788270062.542361 mapwrite 0x02aa08 thru 0x02f8db (0x4ed4 bytes) 000720 1788270062.571128 write 0x0318ef thru 0x0335f5 (0x1d07 bytes) 000721 1788270062.572175 fallocate 0x017d29 thru 0x025af4 (0xddcc bytes) 000722 1788270062.577147 mapread 0x00338f thru 0x006ccd (0x393f bytes) 000724 1788270062.582343 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x335f6): Mirror 12: 0xb5322237 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x335f6): Mirror 12: 0xb5322237 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x335f6: skipped 000725 1788270062.771977 fallocate 0x01dbb6 thru 0x01f4c2 (0x190d bytes) 000726 1788270062.774894 trunc from 0x017901 to 0x01ee15 (0x7515 bytes) 000727 1788270062.777877 write 0x02634b thru 0x02f867 (0x951d bytes) 000728 1788270062.788572 fallocate 0x031fc1 thru 0x0397cc (0x780c bytes) 000729 1788270062.792219 mapread 0x029305 thru 0x0335f5 (0xa2f1 bytes) 000730 1788270062.803023 write 0x00fa15 thru 0x01bce5 (0xc2d1 bytes) 000731 1788270062.812642 trunc from 0x0335f6 to 0x00c839 (0x26dbc bytes) 000733 1788270062.821552 trunc from 0x00c83a to 0x02d0be (0x20885 bytes) 000734 1788270062.830970 punch from 0x012ca9 to 0x017daf (0x5107 bytes) 000735 1788270062.833544 fallocate 0x03b315 thru 0x03ffff (0x4ceb bytes) 000736 1788270062.837551 mapwrite 0x0011ad thru 0x011061 (0xfeb5 bytes) 000737 1788270062.863625 mapwrite 0x028c43 thru 0x0382eb (0xf6a9 bytes) 000738 1788270062.897421 trunc from 0x02960f to 0x034d6c (0xb75e bytes) 000739 1788270062.900416 mapread 0x0175c1 thru 0x01962d (0x206d bytes) 000740 1788270062.906343 trunc from 0x00e243 to 0x0104a4 (0x2262 bytes) 000741 1788270062.912564 MIRROR_RESYNC 2 mirrors 000742 1788270063.288151 fallocate 0x02701a thru 0x02ff3c (0x8f23 bytes) 000743 1788270063.301329 punch from 0x00cb2b to 0x00f0f7 (0x25cd bytes) 000745 1788270063.310399 trunc from 0x0382ec to 0x017ef9 (0x203f2 bytes) 000746 1788270063.322109 write 0x01753f thru 0x025f50 (0xea12 bytes) 000747 1788270063.326244 write 0x0184a1 thru 0x027b1b (0xf67b bytes) 000748 1788270063.327573 trunc from 0x010c92 to 0x0146ae (0x3a1d bytes) 000749 1788270063.331290 write 0x028715 thru 0x02eec8 (0x67b4 bytes) 000750 1788270063.332531 fallocate 0x030601 thru 0x03863e (0x803e bytes) 000751 1788270063.334946 mapwrite 0x026f69 thru 0x02f08d (0x8125 bytes) 000752 1788270063.356741 read 0x0115b2 thru 0x01e4c8 (0xcf17 bytes) 000753 1788270063.361738 write 0x004d06 thru 0x00f912 (0xac0d bytes) 000754 1788270063.368159 read 0x00f60b thru 0x010e32 (0x1828 bytes) 000755 1788270063.374103 write 0x0059f5 thru 0x009324 (0x3930 bytes) 000756 1788270063.377117 trunc from 0x02f08e to 0x030d58 (0x1ccb bytes) 000757 1788270063.386198 mapread 0x01d911 thru 0x02c199 (0xe889 bytes) 000758 1788270063.387816 write 0x005450 thru 0x013ee9 (0xea9a bytes) 000759 1788270063.389697 read 0x024a92 thru 0x02bcda (0x7249 bytes) 000760 1788270063.390307 trunc from 0x0236c1 to 0x030d58 (0xd698 bytes) 000761 1788270063.394649 mapwrite 0x0334cf thru 0x034845 (0x1377 bytes) 000763 1788270063.422586 trunc from 0x022661 to 0x0308f5 (0xe295 bytes) 000764 1788270063.425562 mapwrite 0x019e96 thru 0x0202ee (0x6459 bytes) 000765 1788270063.437389 MIRROR_RESYNC 2 mirrors 000767 1788270063.779132 write 0x022a36 thru 0x028476 (0x5a41 bytes) 000768 1788270063.808311 mapwrite 0x0112a6 thru 0x015926 (0x4681 bytes) 000769 1788270063.825926 write 0x00026c thru 0x009123 (0x8eb8 bytes) 000770 1788270063.834727 write 0x02fc2c thru 0x037697 (0x7a6c bytes) 000771 1788270063.838942 fallocate 0x034632 thru 0x03d066 (0x8a35 bytes) 000772 1788270063.841692 punch from 0x0081ab to 0x00a767 (0x25bd bytes) 000773 1788270063.849423 trunc from 0x020f48 to 0x022bc8 (0x1c81 bytes) 000774 1788270063.851811 read 0x01b875 thru 0x02336c (0x7af8 bytes) 000775 1788270063.856361 trunc from 0x03d067 to 0x021161 (0x1bf05 bytes) 000776 1788270063.873116 punch from 0x00acce to 0x01a325 (0xf658 bytes) 000777 1788270063.877528 fallocate 0x0341f9 thru 0x039d93 (0x5b9b bytes) 000778 1788270063.880253 write 0x026ed0 thru 0x0325e3 (0xb714 bytes) 000779 1788270063.881712 mapread 0x023171 thru 0x026a7b (0x390b bytes) 000780 1788270063.884724 read 0x028da9 thru 0x02ba5e (0x2cb6 bytes) 000781 1788270063.887369 MIRROR_RESYNC 2 mirrors 000784 1788270064.345313 fallocate 0x006df2 thru 0x008c39 (0x1e48 bytes) 000785 1788270064.369161 MIRROR_RESYNC 2 mirrors 000786 1788270064.774462 punch from 0x0301c6 to 0x0325e3 (0x241e bytes) 000787 1788270064.802689 punch from 0x00dc12 to 0x01295d (0x4d4c bytes) 000788 1788270064.805562 fallocate 0x006d27 thru 0x0146d2 (0xd9ac bytes) 000789 1788270064.808963 fallocate 0x0060de thru 0x0103d6 (0xa2f9 bytes) 000790 1788270064.812340 mapwrite 0x000541 thru 0x009e05 (0x98c5 bytes) 000791 1788270064.856031 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x325e4): Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x325e4): Mirror 12: 0x39b44d93 (stale) CRC-32 checksum value for chunk [0, 0x325e4): Mirror 1: 0xcc247453 Mirror 13: 0xcc247453 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x325e4: skipped 000793 1788270065.381550 MIRROR_RESYNC 3 mirrors 000794 1788270065.751026 MIRROR_RESYNC 3 mirrors 000795 1788270065.928092 fallocate 0x0044d8 thru 0x00e8f2 (0xa41b bytes) 000796 1788270065.956909 fallocate 0x012495 thru 0x016c4b (0x47b7 bytes) 000797 1788270065.961928 write 0x017114 thru 0x01d5ed (0x64da bytes) 000798 1788270065.973012 trunc from 0x0325e4 to 0x00b47a (0x27169 bytes) 000799 1788270065.996305 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb47b): Mirror 13: 0x5787efd3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb47b): Mirror 13: 0x5787efd3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb47b: skipped 000800 1788270066.171499 fallocate 0x0087c8 thru 0x014bab (0xc3e4 bytes) 000802 1788270066.175909 mapread 0x0019ab thru 0x00b47a (0x9ad0 bytes) 000803 1788270066.187263 mapread 0x009a31 thru 0x00b47a (0x1a4a bytes) 000804 1788270066.188539 fallocate 0x039ef1 thru 0x03ffff (0x610f bytes) 000805 1788270066.195291 MIRROR_RESYNC 2 mirrors 000806 1788270066.502698 trunc from 0x03210a to 0x038fc1 (0x6eb8 bytes) 000807 1788270066.519261 punch from 0x01c6d3 to 0x025238 (0x8b66 bytes) 000808 1788270066.523090 read 0x0328e7 thru 0x03ffff (0xd719 bytes) 000809 1788270066.527792 fallocate 0x01551b thru 0x015b51 (0x637 bytes) 000810 1788270066.531811 mapwrite 0x032d62 thru 0x033cb9 (0xf58 bytes) 000811 1788270066.540915 punch from 0x02f0ec to 0x036183 (0x7098 bytes) 000812 1788270066.543705 mapread 0x01e396 thru 0x029d64 (0xb9cf bytes) 000813 1788270066.563597 fallocate 0x010b49 thru 0x018523 (0x79db bytes) 000814 1788270066.569278 fallocate 0x000e3b thru 0x008e9e (0x8064 bytes) 000815 1788270066.572202 punch from 0x03b370 to 0x03fd99 (0x4a2a bytes) 000817 1788270066.575772 write 0x027d63 thru 0x03763f (0xf8dd bytes) 000818 1788270066.579333 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) 000819 1788270066.676392 punch from 0x03a150 to 0x03ffff (0x5eb0 bytes) 000820 1788270066.681544 mapread 0x01624e thru 0x020541 (0xa2f4 bytes) 000821 1788270066.701607 mapwrite 0x0330d8 thru 0x03f80b (0xc734 bytes) 000822 1788270066.745350 trunc from 0x040000 to 0x006546 (0x39ab9 bytes) 000823 1788270066.758525 fallocate 0x03bd29 thru 0x03f8e2 (0x3bba bytes) 000824 1788270066.762380 trunc from 0x006547 to 0x00c715 (0x61cf bytes) 000825 1788270066.774616 mapwrite 0x02eb41 thru 0x03554c (0x6a0c bytes) 000827 1788270066.799144 read 0x02ef02 thru 0x03554c (0x664b bytes) 000828 1788270066.799670 read 0x02792a thru 0x033e0a (0xc4e1 bytes) 000829 1788270066.800234 trunc from 0x007dee to 0x012fc1 (0xb1d4 bytes) 000830 1788270066.803588 mapwrite 0x026641 thru 0x035312 (0xecd2 bytes) 000831 1788270066.847488 MIRROR_EXTEND to 2 mirrors 000832 1788270067.223226 write 0x00cd6e thru 0x011e72 (0x5105 bytes) 000833 1788270067.250439 mapread 0x014dc8 thru 0x01e95c (0x9b95 bytes) 000834 1788270067.264921 trunc from 0x00d052 to 0x01a7fd (0xd7ac bytes) 000835 1788270067.276710 mapwrite 0x02be38 thru 0x03b775 (0xf93e bytes) 000836 1788270067.335725 read 0x02d22d thru 0x039bad (0xc981 bytes) 000837 1788270067.336477 punch from 0x010641 to 0x013051 (0x2a11 bytes) 000838 1788270067.342620 MIRROR_RESYNC 2 mirrors 000839 1788270067.645559 mapwrite 0x01a79d thru 0x022905 (0x8169 bytes) 000841 1788270067.713360 read 0x022768 thru 0x025abe (0x3357 bytes) 000843 1788270067.720304 trunc from 0x03b776 to 0x02df08 (0xd86d bytes) 000844 1788270067.751772 punch from 0x006d47 to 0x00a9e4 (0x3c9e bytes) 000845 1788270067.755385 punch from 0x013ebb to 0x0149a5 (0xaeb bytes) 000846 1788270067.767377 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) 000847 1788270067.891120 MIRROR_EXTEND to 2 mirrors 000848 1788270068.283241 write 0x02da12 thru 0x032170 (0x475f bytes) 000849 1788270068.309702 mapwrite 0x030fb6 thru 0x036607 (0x5652 bytes) 000851 1788270068.351508 write 0x00ffce thru 0x012457 (0x248a bytes) 000853 1788270068.358525 mapread 0x008b35 thru 0x011b45 (0x9011 bytes) 000854 1788270068.364046 mapwrite 0x02eee0 thru 0x0342fe (0x541f bytes) 000855 1788270068.375297 fallocate 0x026a4d thru 0x0367ae (0xfd62 bytes) 000856 1788270068.384059 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) 000857 1788270068.482623 fallocate 0x00b063 thru 0x010ced (0x5c8b bytes) 000858 1788270068.487876 trunc from 0x02dfa1 to 0x03456b (0x65cb bytes) 000860 1788270068.494253 read 0x01c772 thru 0x029e39 (0xd6c8 bytes) 000861 1788270068.501263 mapread 0x009998 thru 0x00a7e0 (0xe49 bytes) 000862 1788270068.507515 mapread 0x00041c thru 0x00515c (0x4d41 bytes) 000863 1788270068.516122 mapread 0x013d42 thru 0x016a91 (0x2d50 bytes) 000864 1788270068.522772 read 0x030f89 thru 0x033d19 (0x2d91 bytes) 000865 1788270068.527915 fallocate 0x01f762 thru 0x02218f (0x2a2e bytes) 000867 1788270068.533150 read 0x0001ab thru 0x00872c (0x8582 bytes) 000869 1788270068.534244 trunc from 0x036608 to 0x027b72 (0xea95 bytes) 000870 1788270068.547087 fallocate 0x03e3fb thru 0x03ffff (0x1c05 bytes) 000871 1788270068.552220 trunc from 0x010fef to 0x01a3ee (0x9400 bytes) 000872 1788270068.557783 punch from 0x0188de to 0x01ba7c (0x319f bytes) 000873 1788270068.561732 fallocate 0x034979 thru 0x03baef (0x7177 bytes) 000874 1788270068.567018 mapread 0x027061 thru 0x02c825 (0x57c5 bytes) 000875 1788270068.576445 fallocate 0x00a6bd thru 0x00e32b (0x3c6f bytes) 000876 1788270068.580329 mapwrite 0x00a3e3 thru 0x012943 (0x8561 bytes) 000877 1788270068.593438 fallocate 0x020d06 thru 0x02387b (0x2b76 bytes) 000878 1788270068.597465 trunc from 0x040000 to 0x0246f5 (0x1b90a bytes) 000880 1788270068.616631 mapread 0x014df0 thru 0x016f7c (0x218d bytes) 000881 1788270068.617692 mapwrite 0x03984b thru 0x03ffff (0x67b5 bytes) 000882 1788270068.676802 mapread 0x038f56 thru 0x03ffff (0x70aa bytes) 000883 1788270068.682034 fallocate 0x0112b2 thru 0x020f0c (0xfc5b bytes) 000884 1788270068.689855 trunc from 0x040000 to 0x02ee42 (0x111bd bytes) 000887 1788270068.710676 trunc from 0x0197bb to 0x020330 (0x6b76 bytes) 000888 1788270068.717072 punch from 0x0242aa to 0x02ee42 (0xab99 bytes) 000889 1788270068.720372 trunc from 0x02ee43 to 0x03ef90 (0x1014e bytes) 000890 1788270068.734551 MIRROR_EXTEND to 2 mirrors 000891 1788270069.152497 read 0x005aa9 thru 0x013e89 (0xe3e1 bytes) 000894 1788270069.183338 trunc from 0x03ef91 to 0x0099e3 (0x355ad bytes) 000895 1788270069.214332 trunc from 0x0099e4 to 0x00561b (0x43c8 bytes) 000896 1788270069.224361 trunc from 0x00561c to 0x03e8c6 (0x392ab bytes) 000897 1788270069.233528 fallocate 0x0386fd thru 0x03a1f5 (0x1af9 bytes) 000898 1788270069.236549 mapwrite 0x0191d3 thru 0x01f59d (0x63cb bytes) 000899 1788270069.262733 read 0x0143c1 thru 0x01c39b (0x7fdb bytes) 000900 1788270069.267152 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e8c7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8c7): Mirror 2: 0x8ca20364 (stale) CRC-32 checksum value for chunk [0, 0x3e8c7): Mirror 1: 0xb6650eae Mirror 3: 0xb6650eae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8c7: skipped 000901 1788270069.757405 trunc from 0x02cb9a to 0x02f1b2 (0x2619 bytes) 000902 1788270069.780583 write 0x0236df thru 0x02f3dc (0xbcfe bytes) 000903 1788270069.793102 mapread 0x0074f5 thru 0x00ef49 (0x7a55 bytes) 000904 1788270069.812506 punch from 0x016316 to 0x01ef22 (0x8c0d bytes) 000905 1788270069.816009 trunc from 0x001ac9 to 0x008106 (0x663e bytes) 000906 1788270069.820802 fallocate 0x033984 thru 0x03ffff (0xc67c bytes) 000907 1788270069.824371 fallocate 0x037c4e thru 0x0387c6 (0xb79 bytes) 000909 1788270069.828209 fallocate 0x006024 thru 0x0063fb (0x3d8 bytes) 000910 1788270069.834597 trunc from 0x00e26d to 0x0184dc (0xa270 bytes) 000911 1788270069.839765 read 0x030530 thru 0x03768e (0x715f bytes) 000912 1788270069.849283 punch from 0x030f6f to 0x03324e (0x22e0 bytes) 000913 1788270069.852695 write 0x02c7ef thru 0x03935f (0xcb71 bytes) 000914 1788270069.861168 MIRROR_RESYNC 3 mirrors 000915 1788270070.340350 punch from 0x00eab6 to 0x01833c (0x9887 bytes) 000918 1788270070.365596 fallocate 0x0293ee thru 0x037d42 (0xe955 bytes) 000919 1788270070.370441 write 0x01490c thru 0x01a283 (0x5978 bytes) 000920 1788270070.377799 write 0x000f10 thru 0x00b50e (0xa5ff bytes) 000921 1788270070.387563 read 0x025547 thru 0x0279b3 (0x246d bytes) 000922 1788270070.393273 fallocate 0x03e910 thru 0x03fc5e (0x134f bytes) 000923 1788270070.396928 mapwrite 0x003949 thru 0x0114c6 (0xdb7e bytes) 000924 1788270070.428107 fallocate 0x00a326 thru 0x00da31 (0x370c bytes) 000925 1788270070.431633 write 0x021f97 thru 0x025055 (0x30bf bytes) 000926 1788270070.436695 fallocate 0x017267 thru 0x01e99e (0x7738 bytes) 000927 1788270070.440939 write 0x0121a2 thru 0x0207a9 (0xe608 bytes) 000928 1788270070.445887 mapwrite 0x000226 thru 0x00ac0a (0xa9e5 bytes) 000929 1788270070.464521 fallocate 0x016e04 thru 0x02392e (0xcb2b bytes) 000930 1788270070.467710 write 0x010db1 thru 0x01789b (0x6aeb bytes) 000931 1788270070.469399 write 0x011234 thru 0x0203fb (0xf1c8 bytes) 000932 1788270070.470597 punch from 0x022c6f to 0x031e3c (0xf1ce bytes) 000933 1788270070.473944 write 0x024bae thru 0x026c63 (0x20b6 bytes) 000934 1788270070.479033 punch from 0x022e97 to 0x0327e1 (0xf94b bytes) 000935 1788270070.489272 fallocate 0x02723d thru 0x02788a (0x64e bytes) 000937 1788270070.492256 fallocate 0x012cb9 thru 0x0159fc (0x2d44 bytes) 000938 1788270070.495367 trunc from 0x004ea3 to 0x00d886 (0x89e4 bytes) 000939 1788270070.498523 read 0x012227 thru 0x01f6c5 (0xd49f bytes) 000940 1788270070.499196 trunc from 0x00d7a0 to 0x00eb82 (0x13e3 bytes) 000941 1788270070.502213 trunc from 0x040000 to 0x01bee5 (0x2411a bytes) 000942 1788270070.524464 fallocate 0x029aa2 thru 0x038c87 (0xf1e6 bytes) 000943 1788270070.530164 fallocate 0x00acb6 thru 0x011515 (0x6860 bytes) 000944 1788270070.533984 mapread 0x00aa1e thru 0x00b98e (0xf71 bytes) 000945 1788270070.545669 MIRROR_RESYNC 3 mirrors 000947 1788270070.835046 trunc from 0x000beb to 0x00da53 (0xce69 bytes) 000948 1788270070.854518 mapread 0x00f636 thru 0x00fec4 (0x88f bytes) 000949 1788270070.860299 fallocate 0x0247df thru 0x028a2f (0x4251 bytes) 000950 1788270070.866174 mapwrite 0x038363 thru 0x03ffff (0x7c9d bytes) 000951 1788270070.932091 write 0x01fb4a thru 0x028c90 (0x9147 bytes) 000952 1788270070.940531 mapread 0x000e0d thru 0x002629 (0x181d bytes) 000953 1788270070.949669 read 0x031841 thru 0x03ffff (0xe7bf bytes) 000954 1788270070.953747 fallocate 0x037e39 thru 0x03ffff (0x81c7 bytes) 000955 1788270070.959128 fallocate 0x0073a5 thru 0x00b0b5 (0x3d11 bytes) 000956 1788270070.962236 fallocate 0x02b955 thru 0x0310ab (0x5757 bytes) 000957 1788270070.965588 trunc from 0x040000 to 0x03efaf (0x1050 bytes) 000958 1788270070.976529 trunc from 0x025757 to 0x032055 (0xc8ff bytes) 000959 1788270070.984627 mapwrite 0x01ee9c thru 0x029437 (0xa59c bytes) 000960 1788270071.001044 mapread 0x03be24 thru 0x03efaf (0x318c bytes) 000961 1788270071.004290 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3efb0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efb0): Mirror 2: 0x92d8ad3a (stale) Mirror 3: 0x92d8ad3a (stale) CRC-32 checksum value for chunk [0, 0x3efb0): Mirror 1: 0xc594014f Mirror 4: 0xc594014f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efb0: skipped 000962 1788270071.462799 fallocate 0x00109b thru 0x009d4a (0x8cb0 bytes) 000964 1788270071.497716 mapwrite 0x029490 thru 0x02b872 (0x23e3 bytes) 000965 1788270071.530152 trunc from 0x03efb0 to 0x025bc4 (0x193eb bytes) 000967 1788270071.548488 mapread 0x001089 thru 0x00904b (0x7fc3 bytes) 000968 1788270071.558952 trunc from 0x025bc5 to 0x03d354 (0x17790 bytes) 000969 1788270071.570608 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d355): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d355): Mirror 2: 0x77ea7fce (stale) Mirror 3: 0x77ea7fce (stale) Mirror 4: 0x38e0fe48 (stale) CRC-32 checksum value for chunk [0, 0x3d355): Mirror 1: 0xfcbf0e40 Mirror 5: 0xfcbf0e40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d355: skipped 000970 1788270072.057402 write 0x03379c thru 0x03f32d (0xbb92 bytes) 000971 1788270072.081819 trunc from 0x025d5e to 0x034eb2 (0xf155 bytes) 000972 1788270072.093572 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f32e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f32e): Mirror 2: 0xbe4ba516 (stale) Mirror 3: 0xbe4ba516 (stale) Mirror 4: 0x6b215750 (stale) Mirror 5: 0xba56eba0 (stale) CRC-32 checksum value for chunk [0, 0x3f32e): Mirror 1: 0x918707ba Mirror 6: 0x918707ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f32e: skipped 000973 1788270072.707906 fallocate 0x01049f thru 0x01c414 (0xbf76 bytes) 000974 1788270072.732627 mapwrite 0x015b22 thru 0x015c92 (0x171 bytes) 000975 1788270072.748621 mapwrite 0x0301de thru 0x03a659 (0xa47c bytes) 000976 1788270072.784659 punch from 0x027fdb to 0x02d83e (0x5864 bytes) 000977 1788270072.787872 trunc from 0x03d1f9 to 0x03f32d (0x2135 bytes) 000978 1788270072.792017 mapread 0x007da3 thru 0x013a5a (0xbcb8 bytes) 000979 1788270072.805873 punch from 0x01d14d to 0x0215e2 (0x4496 bytes) 000980 1788270072.810918 mapwrite 0x009946 thru 0x011236 (0x78f1 bytes) 000981 1788270072.825620 fallocate 0x03dd94 thru 0x03ffff (0x226c bytes) 000982 1788270072.828953 punch from 0x002667 to 0x00509f (0x2a39 bytes) 000983 1788270072.835641 fallocate 0x00e49b thru 0x0132c3 (0x4e29 bytes) 000985 1788270072.840337 mapwrite 0x012799 thru 0x022437 (0xfc9f bytes) 000986 1788270072.862368 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f32e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f32e): Mirror 2: 0xbe4ba516 (stale) Mirror 3: 0xbe4ba516 (stale) Mirror 4: 0x6b215750 (stale) Mirror 5: 0xba56eba0 (stale) Mirror 6: 0x918707ba (stale) CRC-32 checksum value for chunk [0, 0x3f32e): Mirror 1: 0x3de687a0 Mirror 7: 0x3de687a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f32e: skipped 000988 1788270073.376197 fallocate 0x0178ad thru 0x01b8f8 (0x404c bytes) 000989 1788270073.404915 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f32e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f32e): Mirror 2: 0xbe4ba516 (stale) Mirror 3: 0xbe4ba516 (stale) Mirror 4: 0x6b215750 (stale) Mirror 5: 0xba56eba0 (stale) Mirror 6: 0x918707ba (stale) Mirror 7: 0x3de687a0 (stale) CRC-32 checksum value for chunk [0, 0x3f32e): Mirror 1: 0x3de687a0 Mirror 8: 0x3de687a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f32e: skipped 000991 1788270073.982581 fallocate 0x004873 thru 0x00c42f (0x7bbd bytes) 000993 1788270074.009976 mapwrite 0x026473 thru 0x026612 (0x1a0 bytes) 000994 1788270074.020445 trunc from 0x010597 to 0x01088c (0x2f6 bytes) 000995 1788270074.023941 trunc from 0x03f32e to 0x027e77 (0x174b6 bytes) 000996 1788270074.038745 mapread 0x007329 thru 0x00fc74 (0x894c bytes) 000997 1788270074.054247 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x354:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27e78): Mirror 3: 0xf3659d68 (stale) Mirror 4: 0xfdad835a (stale) Mirror 5: 0xbfd9de25 (stale) Mirror 6: 0xbfd9de25 (stale) Mirror 7: 0x4a94b284 (stale) Mirror 8: 0x4a94b284 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27e78): Mirror 3: 0xf3659d68 (stale) Mirror 4: 0xfdad835a (stale) Mirror 5: 0xbfd9de25 (stale) Mirror 6: 0xbfd9de25 (stale) Mirror 7: 0x4a94b284 (stale) Mirror 8: 0x4a94b284 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27e78: skipped 000998 1788270074.267571 write 0x02da41 thru 0x03bc68 (0xe228 bytes) 000999 1788270074.271594 read 0x01aeac thru 0x01e365 (0x34ba bytes) 001000 1788270074.276042 mapwrite 0x02c3ee thru 0x030ae9 (0x46fc bytes) All operations completed A-OK! PASS 70a (48s) == sanity-flr test 71: check mirror extend parent fid ==== 09:41:16 (1788270076) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x391:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (5s) == sanity-flr test 72: check append on FLR file ========== 09:41:21 (1788270081) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x108:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0820907 s, 25.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x108:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635348 s, 16.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x109:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x108:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x109:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x108:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 09:41:24 (1788270084) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2080896 .. Extending file size to 3683712 .. Extending file size to 8227936 .. Extending file size to 6260448 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7243872 .. failed Extending file size to 8164288 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8401984 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8490592 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8544576 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8961120 .. Extending file size to 9088960 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9090208 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9102368 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62322 62323 62325 62327 62328 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) 192.168.203.116@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:42:32 (1788270152) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2290880 .. Extending file size to 7331232 .. Extending file size to 7644928 .. Extending file size to 5927360 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8776000 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8202592 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8994304 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9180224 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9230528 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9313440 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 65078 65079 65081 65082 65084 failed 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) 192.168.203.116@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:43:38 (1788270218) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1268828 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2416 1268888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24940 3582080 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19032 3587988 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43972 7170068 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1268828 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2416 1268888 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24940 3582080 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 19032 3587988 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43972 7170068 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0472046 s, 44.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0591614 s, 35.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x7ccedac6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:43:45 (1788270225) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0463806 s, 45.2 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 09:43:48 (1788270228) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x38:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x122:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x122:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x123:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x11e:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:43:50 (1788270230) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0416348 s, 50.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x11f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x120:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:43:53 (1788270233) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.02918 s, 71.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x121:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x121:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:43:56 (1788270236) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0317899 s, 66.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47d:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:44:21 (1788270261) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449054 s, 23.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47e:0x0] } PASS 204d (32s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:44:53 (1788270293) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10363 s, 20.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (42s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:45:35 (1788270335) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg316-server: Pool lustre.test_204f created Adding targets to pool oleg316-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg316-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124497 s, 16.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg316-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg316-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg316-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:46:50 (1788270410) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:46:54 (1788270414) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:46:58 (1788270418) oleg316-server: Pool lustre.test_206 created Adding targets to pool oleg316-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg316-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg316-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg316-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg316-server: Pool lustre.test_206 destroyed PASS 206 (20s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:47:18 (1788270438) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043345 s, 24.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361918 s, 29.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1c88d15e Mirror 3: 0x1c88d15e PASS 207 (6s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:47:24 (1788270444) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:47:26 (1788270446) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:47:29 (1788270449) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0380623 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1a50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x9b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x9b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x9b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x10a0000 Read this (4 bytes): BBBB 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) PASS 209a (24s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:47:53 (1788270473) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0124328 s, 659 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:47:58 (1788270478) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412189 s, 25.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59 lmm_fid: [0x200000405:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59 lmm_fid: [0x200000405:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611225 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:47:59.000000000 -0400 Modify: 2026-09-01 09:47:59.000000000 -0400 Change: 2026-09-01 09:48:00.000000000 -0400 Birth: 2026-09-01 09:47:59.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:48:03 (1788270483) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345383 s, 30.4 MB/s oleg316-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg316-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg316-server: O/280000401/d16/130: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:48:17 (1788270497) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.110219 s, 372 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0149738 s, 274 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 1 09:48 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 1 09:48 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:48:22 (1788270502) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:48:26 (1788270506) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1120 sec =========== 09:48:33 (1788270513) === sanity-flr: start cleanup 09:48:34 (1788270514) === === sanity-flr: finish cleanup 09:48:41 (1788270521) ===