-----============= acceptance-small: sanity-flr ============----- Mon Aug 17 00:07:50 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg133-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 00:08:00 (1786939680) === oleg133-client.virtnet: executing check_config_client /mnt/lustre oleg133-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg133-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa04082bf8000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa04082bf8000.idle_timeout=debug disable quota as required oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 00:08:19 (1786939699) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8202] == sanity-flr test 0a: lfs mirror create with -N option == 00:08:20 (1786939700) PASS 0a (9s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 00:08:29 (1786939709) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 00:08:31 (1786939711) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 00:08:33 (1786939713) PASS 0d (15s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 00:08:48 (1786939728) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 00:08:50 (1786939730) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 00:08:51 (1786939731) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa04082bf8000.stats=clear osc.lustre-OST0001-osc-ffffa04082bf8000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 00:08:56 (1786939736) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 00:09:01 (1786939741) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 00:09:05 (1786939745) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 00:09:07 (1786939747) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 00:09:11 (1786939751) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 00:09:14 (1786939754) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 00:09:17 (1786939757) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 00:09:21 (1786939761) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 00:09:24 (1786939764) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 00:09:29 (1786939769) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 00:09:32 (1786939772) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.710978 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493835 s, 21.2 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 00:09:37 (1786939777) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168495 s, 62.2 MB/s PASS 22 (3s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 00:09:40 (1786939780) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 00:09:45 (1786939785) Stopping /mnt/lustre-ost1 (opts:) on oleg133-server oleg133-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa04082bf8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa04082bf8000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg133-server: oleg133-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg133-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg133-server oleg133-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa04082bf8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa04082bf8000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg133-server: oleg133-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg133-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (36s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 00:10:21 (1786939821) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0638299 s, 32.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg133-server oleg133-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa04082bf8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa04082bf8000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg133-server: oleg133-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg133-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (18s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 00:10:40 (1786939840) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg133-server oleg133-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa04082bf8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa04082bf8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg133-server: oleg133-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg133-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg133-server oleg133-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa04082bf8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa04082bf8000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg133-server: oleg133-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg133-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:11:05 (1786939865) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.898017 s, 58.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 17 00:11 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.229s user 0m0.002s sys 0m0.148s Stopping /mnt/lustre-ost1 (opts:) on oleg133-server oleg133-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa04082bf8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa04082bf8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.256s user 0m0.002s sys 0m0.149s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg133-server: oleg133-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg133-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg133-server oleg133-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa04082bf8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa04082bf8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.319s user 0m0.000s sys 0m0.150s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg133-server: oleg133-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg133-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg133-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa04082bf8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa04082bf8000.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:11:34 (1786939894) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:11:35 (1786939895) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:11:36 (1786939896) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 00:11:37 (1786939897) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 00:11:39 (1786939899) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.154451 s, 81.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:11:42 (1786939902) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125113 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108318 s, 96.8 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 00:11:44 (1786939904) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 00:11:47 (1786939907) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000709631 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000495917 s, 2.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000622932 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00061691 s, 1.7 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 00:11:50 (1786939910) create test files with size 15732010 .. d0ad68dc47dc190a36b031d30a58866d - 8b9f8d52be3e0cd4b716a0f6131fea80 - d0bc0076474c7a9fa0092bd5f406ed29 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 00:11:55 (1786939915) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 00:11:59 (1786939919) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 00:12:01 (1786939921) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253483 s, 41.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc5:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138891 s, 75.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xab:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 00:12:04 (1786939924) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0319239 s, 65.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.112927 s, 37.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0494476 s, 42.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 00:12:08 (1786939928) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 00:12:09 (1786939929) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 00:12:10 (1786939930) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103773 s, 101 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 00:12:12 (1786939932) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0748367 s, 42.0 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 00:12:15 (1786939935) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 00:12:18 (1786939938) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0955262 s, 110 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (19s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 00:12:37 (1786939957) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa04087c72000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0779779 s, 134 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa040853a6800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 00:12:41 (1786939961) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 00:12:52 (1786939972) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 00:13:03 (1786939983) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 00:13:04 (1786939984) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0913375 s, 34.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0600039 s, 52.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x91cd2964 (nosync,stale) Mirror 2: 0x91cd2964 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 1b54f7e1b2cbea70290e78a24aa1ab5a - mirror 1 checksum: 0c8c1354b7a8ad6a8039dfe424a216f1 - mirror 2 checksum: 1b54f7e1b2cbea70290e78a24aa1ab5a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T00:13:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T00:13:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1b54f7e1b2cbea70290e78a24aa1ab5a - mirror 1 checksum: 0c8c1354b7a8ad6a8039dfe424a216f1 - mirror 2 checksum: 0c8c1354b7a8ad6a8039dfe424a216f1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T00:13:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T00:13:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 00:13:07 (1786939987) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179351 s, 58.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 00:13:10 (1786939990) Failing mds1 on oleg133-server Stopping /mnt/lustre-mds1 (opts:) on oleg133-server 00:13:18 (1786939998) shut down facet: mds1 facet_host: oleg133-server facet_failover_host: oleg133-server Failover mds1 to oleg133-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 00:13:32 (1786940012) targets are mounted 00:13:32 (1786940012) facet_failover done oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113426 s, 92.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 00:13:41 (1786940021) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00922326 s, 2.2 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00424184 s, 4.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 00:13:43 (1786940023) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00872348 s, 2.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00339428 s, 6.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 00:13:46 (1786940026) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00261223 s, 6.3 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 00:13:48 (1786940028) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0113318 s, 1.8 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 00:13:51 (1786940031) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00198749 s, 2.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00721411 s, 568 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 00:13:53 (1786940033) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 00:13:59 (1786940039) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:14:18 (1786940058) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 00:14:35 (1786940075) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 00:14:52 (1786940092) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 00:14:54 (1786940094) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57430 mirror split pid 57431 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57430 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57431 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 00:15:57 (1786940157) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 157 truncating to largest ever: 0x2fd05 000001 1786940157.708281 trunc from 00000000 to 0x02fd04 (0x2fd05 bytes) 000002 1786940157.712050 mapread 0x014a1d thru 0x015d73 (0x1357 bytes) 000003 1786940157.715062 fallocate 0x0238bd thru 0x02af2f (0x7673 bytes) fallocating to largest ever: 0x40000 000004 1786940157.716592 fallocate 0x039c94 thru 0x03ffff (0x636c bytes) 000005 1786940157.718266 fallocate 0x02c61a thru 0x036447 (0x9e2e bytes) 000006 1786940157.719980 read 0x00bb2b thru 0x014e69 (0x933f bytes) 000007 1786940157.722755 fallocate 0x00b1d3 thru 0x01261a (0x7448 bytes) 000008 1786940157.724148 write 0x0279b4 thru 0x028e10 (0x145d bytes) 000009 1786940157.726979 punch from 0x01748e to 0x019999 (0x250c bytes) 000010 1786940157.728438 write 0x018b1f thru 0x01e244 (0x5726 bytes) 000011 1786940157.731258 mapwrite 0x01799a thru 0x0250f5 (0xd75c bytes) 000013 1786940157.743673 mapwrite 0x03ea46 thru 0x03ffff (0x15ba bytes) 000014 1786940157.752784 write 0x01e838 thru 0x027e8d (0x9656 bytes) 000016 1786940157.753997 trunc from 0x040000 to 0x0067f1 (0x3980e bytes) 000017 1786940157.757808 mapread 0x0067a0 thru 0x0067f1 (0x52 bytes) 000018 1786940157.760527 fallocate 0x023dc4 thru 0x0322c8 (0xe505 bytes) 000019 1786940157.762175 fallocate 0x03fabd thru 0x03ffff (0x543 bytes) 000020 1786940157.763724 punch from 0x00312b to 0x00f100 (0xbfd6 bytes) 000021 1786940157.765300 trunc from 0x0322c9 to 0x02923b (0x908d bytes) 000022 1786940157.769903 write 0x01b938 thru 0x01e99e (0x3067 bytes) 000023 1786940157.772512 mapread 0x023a85 thru 0x024e35 (0x13b1 bytes) 000024 1786940157.775225 read 0x01661c thru 0x01af3d (0x4922 bytes) 000026 1786940157.778591 trunc from 0x02923c to 0x001818 (0x27a23 bytes) 000028 1786940157.783097 trunc from 0x0005ef to 0x001818 (0x122a bytes) 000029 1786940157.784552 write 0x005502 thru 0x013d47 (0xe846 bytes) 000031 1786940157.786165 write 0x025b89 thru 0x031a94 (0xbf0c bytes) 000032 1786940157.787017 mapwrite 0x02ed48 thru 0x036737 (0x79f0 bytes) 000033 1786940157.800923 MIRROR_EXTEND to 2 mirrors 000034 1786940157.945888 MIRROR_RESYNC 2 mirrors 000035 1786940158.038399 trunc from 0x010f23 to 0x01be6a (0xaf48 bytes) 000036 1786940158.048890 read 0x01ad68 thru 0x02639e (0xb637 bytes) 000037 1786940158.052052 fallocate 0x02ae0a thru 0x034d62 (0x9f59 bytes) 000038 1786940158.054092 write 0x00c839 thru 0x01136c (0x4b34 bytes) 000041 1786940158.059746 punch from 0x031379 to 0x035deb (0x4a73 bytes) 000042 1786940158.061871 read 0x022bd6 thru 0x02a4df (0x790a bytes) 000043 1786940158.065017 trunc from 0x036738 to 0x0088b9 (0x2de7e bytes) 000044 1786940158.076905 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) 000046 1786940158.114333 MIRROR_EXTEND to 2 mirrors 000047 1786940158.253227 mapwrite 0x0144c9 thru 0x019111 (0x4c49 bytes) 000048 1786940158.275457 trunc from 0x001631 to 0x00be1d (0xa7ed bytes) 000049 1786940158.277231 trunc from 0x00b1fd to 0x0130b2 (0x7eb6 bytes) 000050 1786940158.278964 trunc from 0x0167bd to 0x019111 (0x2955 bytes) 000051 1786940158.280433 write 0x014b3a thru 0x017367 (0x282e bytes) 000052 1786940158.282080 fallocate 0x007cbb thru 0x00e4f2 (0x6838 bytes) 000053 1786940158.283568 trunc from 0x019112 to 0x01770e (0x1a03 bytes) 000054 1786940158.288149 mapread 0x00b765 thru 0x00c384 (0xc20 bytes) 000055 1786940158.290839 trunc from 0x01770f to 0x028e57 (0x11749 bytes) 000056 1786940158.294373 write 0x01c470 thru 0x01fbc3 (0x3754 bytes) 000057 1786940158.296878 trunc from 0x028e58 to 0x0292f8 (0x4a1 bytes) 000058 1786940158.300590 mapwrite 0x02891a thru 0x02da56 (0x513d bytes) 000059 1786940158.311856 trunc from 0x02da57 to 0x00c561 (0x214f5 bytes) 000060 1786940158.315857 punch from 0x001319 to 0x00c561 (0xb249 bytes) 000061 1786940158.317425 mapread 0x004239 thru 0x00b32f (0x70f7 bytes) 000062 1786940158.320508 write 0x033092 thru 0x03971e (0x668d bytes) 000064 1786940158.321832 write 0x0103be thru 0x0182b0 (0x7ef3 bytes) 000065 1786940158.324570 trunc from 0x013468 to 0x016e82 (0x3a1b bytes) 000066 1786940158.328449 punch from 0x030212 to 0x03971e (0x950d bytes) 000067 1786940158.329820 trunc from 0x03971f to 0x01dd79 (0x1b9a5 bytes) 000068 1786940158.333694 trunc from 0x00570f to 0x00f962 (0xa254 bytes) 000069 1786940158.335485 mapread 0x01a4f4 thru 0x01dd79 (0x3886 bytes) 000070 1786940158.338697 trunc from 0x01dd7a to 0x027647 (0x98ce bytes) 000071 1786940158.344244 MIRROR_RESYNC 2 mirrors 000072 1786940158.515835 fallocate 0x0384a7 thru 0x03ffff (0x7b59 bytes) 000073 1786940158.524458 read 0x019361 thru 0x0195b1 (0x251 bytes) 000074 1786940158.526228 mapread 0x031f6b thru 0x03df25 (0xbfbb bytes) 000075 1786940158.532914 write 0x00ade8 thru 0x019bd5 (0xedee bytes) 000076 1786940158.535454 trunc from 0x00c71b to 0x013b02 (0x73e8 bytes) 000077 1786940158.539337 mapwrite 0x011b43 thru 0x021664 (0xfb22 bytes) 000078 1786940158.552148 mapwrite 0x00b09f thru 0x012791 (0x76f3 bytes) 000079 1786940158.557923 mapwrite 0x02ac2a thru 0x02b3c4 (0x79b bytes) 000080 1786940158.564017 mapread 0x03b465 thru 0x03ffff (0x4b9b bytes) 000081 1786940158.564916 mapwrite 0x015e36 thru 0x01dc1d (0x7de8 bytes) 000082 1786940158.570859 fallocate 0x0064b8 thru 0x00eb37 (0x8680 bytes) 000083 1786940158.572503 read 0x02a043 thru 0x032a37 (0x89f5 bytes) 000084 1786940158.572936 write 0x010deb thru 0x01ac29 (0x9e3f bytes) 000085 1786940158.573818 write 0x02cf7f thru 0x035c0c (0x8c8e bytes) 000086 1786940158.574611 mapread 0x01e76f thru 0x02a228 (0xbaba bytes) 000087 1786940158.578657 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) 000088 1786940158.611705 write 0x000aab thru 0x0054ae (0x4a04 bytes) 000089 1786940158.615112 punch from 0x0340e5 to 0x03cf30 (0x8e4c bytes) 000090 1786940158.616605 mapread 0x021a19 thru 0x02a344 (0x892c bytes) 000091 1786940158.621219 write 0x004f1c thru 0x00f12b (0xa210 bytes) 000093 1786940158.623831 mapread 0x03f598 thru 0x03ffff (0xa68 bytes) 000094 1786940158.626088 write 0x012e67 thru 0x020a20 (0xdbba bytes) 000095 1786940158.628653 write 0x031795 thru 0x03b695 (0x9f01 bytes) 000096 1786940158.631199 punch from 0x0358db to 0x03ffff (0xa725 bytes) 000097 1786940158.635798 punch from 0x024035 to 0x030eae (0xce7a bytes) 000098 1786940158.637695 mapread 0x0053ea thru 0x00cd1a (0x7931 bytes) 000099 1786940158.638810 punch from 0x00e124 to 0x013176 (0x5053 bytes) 000100 1786940158.640570 trunc from 0x040000 to 0x02ff47 (0x100b8 bytes) 000101 1786940158.648593 trunc from 0x0048fe to 0x00a3c3 (0x5ac6 bytes) 000102 1786940158.650254 trunc from 0x02ff48 to 0x01850c (0x17a3b bytes) 000104 1786940158.654671 fallocate 0x00b15e thru 0x00cc75 (0x1b18 bytes) 000106 1786940158.657476 write 0x0210d2 thru 0x02b3a3 (0xa2d2 bytes) 000107 1786940158.658585 trunc from 0x02b3a4 to 0x0028ec (0x28ab7 bytes) 000108 1786940158.662718 trunc from 0x000122 to 0x00274b (0x262a bytes) 000109 1786940158.664354 mapread 0x000f65 thru 0x0028ec (0x1988 bytes) 000110 1786940158.666340 trunc from 0x0028ed to 0x01d4c8 (0x1abdc bytes) 000111 1786940158.670218 trunc from 0x01d4c9 to 0x010503 (0xcfc5 bytes) 000112 1786940158.675064 MIRROR_EXTEND to 2 mirrors 000113 1786940158.839881 mapread 0x004127 thru 0x008e7d (0x4d57 bytes) 000114 1786940158.844859 punch from 0x00bb82 to 0x010503 (0x4982 bytes) 000115 1786940158.854387 write 0x021ea5 thru 0x0289ce (0x6b2a bytes) 000116 1786940158.856319 mapwrite 0x0306e9 thru 0x037917 (0x722f bytes) 000117 1786940158.873568 punch from 0x01c654 to 0x0275f2 (0xaf9f bytes) 000118 1786940158.875385 mapread 0x00cf2e thru 0x00d977 (0xa4a bytes) 000119 1786940158.878273 trunc from 0x037918 to 0x009203 (0x2e714 bytes) 000120 1786940158.882234 punch from 0x002c80 to 0x0062a6 (0x3627 bytes) 000121 1786940158.883973 read 0x007366 thru 0x0083ef (0x108a bytes) 000122 1786940158.884580 punch from 0x0025a2 to 0x009203 (0x6c62 bytes) 000123 1786940158.886183 write 0x008c59 thru 0x00ffa8 (0x7350 bytes) 000124 1786940158.888281 mapwrite 0x03641a thru 0x03ffff (0x9be6 bytes) 000125 1786940158.904640 fallocate 0x01cb0b thru 0x022ef1 (0x63e7 bytes) 000126 1786940158.906636 trunc from 0x012648 to 0x015153 (0x2b0c bytes) 000127 1786940158.908220 read 0x003691 thru 0x00f366 (0xbcd6 bytes) 000128 1786940158.912011 trunc from 0x040000 to 0x03ac91 (0x536e bytes) 000129 1786940158.916701 fallocate 0x0012d0 thru 0x00f308 (0xe039 bytes) 000130 1786940158.918882 read 0x026f53 thru 0x02f762 (0x8810 bytes) 000131 1786940158.919531 read 0x016310 thru 0x0225de (0xc2cf bytes) 000132 1786940158.920171 trunc from 0x03ac92 to 0x02f622 (0xb66f bytes) 000133 1786940158.925160 punch from 0x018aff to 0x01e8f7 (0x5df9 bytes) 000134 1786940158.927420 fallocate 0x00effd thru 0x013fa2 (0x4fa6 bytes) 000135 1786940158.929568 punch from 0x027d44 to 0x028d68 (0x1025 bytes) 000136 1786940158.931242 fallocate 0x01af2a thru 0x020729 (0x5800 bytes) 000137 1786940158.933167 mapread 0x00def2 thru 0x013458 (0x5567 bytes) 000138 1786940158.935700 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) 000139 1786940158.973019 trunc from 0x02f623 to 0x02fae6 (0x4c4 bytes) 000140 1786940158.977902 fallocate 0x029f12 thru 0x032a31 (0x8b20 bytes) 000141 1786940158.980041 punch from 0x00b8b3 to 0x013ca8 (0x83f6 bytes) 000142 1786940158.981907 mapread 0x016703 thru 0x0246a2 (0xdfa0 bytes) 000143 1786940158.990304 mapwrite 0x03f198 thru 0x03ffff (0xe68 bytes) 000144 1786940158.999968 punch from 0x02a555 to 0x035b11 (0xb5bd bytes) 000145 1786940159.001499 punch from 0x02b11b to 0x031884 (0x676a bytes) 000146 1786940159.002859 fallocate 0x01ac95 thru 0x01bbd4 (0xf40 bytes) 000148 1786940159.004771 fallocate 0x03e57b thru 0x03ffff (0x1a85 bytes) 000149 1786940159.006428 mapwrite 0x031cb3 thru 0x034f4d (0x329b bytes) 000150 1786940159.013065 write 0x0092e3 thru 0x0180a8 (0xedc6 bytes) 000151 1786940159.015677 read 0x02865e thru 0x034980 (0xc323 bytes) 000152 1786940159.016411 mapwrite 0x035e1f thru 0x03749e (0x1680 bytes) 000153 1786940159.022046 fallocate 0x0079dd thru 0x016f78 (0xf59c bytes) 000154 1786940159.023965 read 0x01f8f1 thru 0x022f08 (0x3618 bytes) 000155 1786940159.024553 trunc from 0x040000 to 0x008077 (0x37f88 bytes) 000156 1786940159.029068 trunc from 0x0032ee to 0x008077 (0x4d8a bytes) 000157 1786940159.030669 trunc from 0x002845 to 0x00676a (0x3f26 bytes) 000158 1786940159.032143 read 0x0075eb thru 0x008077 (0xa8d bytes) 000160 1786940159.035417 write 0x009c78 thru 0x00cc6a (0x2ff3 bytes) 000161 1786940159.036324 read 0x001580 thru 0x00cc6a (0xb6eb bytes) 000163 1786940159.037696 trunc from 0x00cc6b to 0x00b4d7 (0x1793 bytes) 000165 1786940159.043031 fallocate 0x0066f5 thru 0x00b37a (0x4c86 bytes) 000166 1786940159.044843 mapread 0x00a5e6 thru 0x00b4d7 (0xef2 bytes) 000167 1786940159.045737 punch from 0x00a099 to 0x00b4d7 (0x143f bytes) 000168 1786940159.048908 punch from 0x006667 to 0x00b4d7 (0x4e71 bytes) 000169 1786940159.050637 write 0x015fb0 thru 0x021d7e (0xbdcf bytes) 000170 1786940159.052106 write 0x0170d9 thru 0x01c4e8 (0x5410 bytes) 000171 1786940159.053061 mapread 0x014ed9 thru 0x021d7e (0xcea6 bytes) 000172 1786940159.056239 mapread 0x015234 thru 0x021d7e (0xcb4b bytes) 000173 1786940159.057243 punch from 0x01a5cd to 0x021d7e (0x77b2 bytes) 000174 1786940159.060905 mapwrite 0x02b422 thru 0x038c89 (0xd868 bytes) 000175 1786940159.075200 mapwrite 0x01c8ec thru 0x021369 (0x4a7e bytes) 000177 1786940159.089602 trunc from 0x02142d to 0x02de59 (0xca2d bytes) 000178 1786940159.091257 mapwrite 0x00fd1f thru 0x01c381 (0xc663 bytes) 000179 1786940159.099381 read 0x01c665 thru 0x01e9a8 (0x2344 bytes) 000180 1786940159.099787 mapread 0x038718 thru 0x038c89 (0x572 bytes) 000181 1786940159.100342 mapread 0x038433 thru 0x038c89 (0x857 bytes) 000182 1786940159.100955 fallocate 0x002e85 thru 0x01194d (0xeac9 bytes) 000183 1786940159.102471 mapread 0x0305d4 thru 0x038c89 (0x86b6 bytes) 000184 1786940159.103148 trunc from 0x030f5f to 0x0368c0 (0x5962 bytes) 000185 1786940159.104688 trunc from 0x038c8a to 0x03ec49 (0x5fc0 bytes) 000187 1786940159.109302 mapwrite 0x016c0c thru 0x021dd7 (0xb1cc bytes) 000188 1786940159.118051 trunc from 0x03ec4a to 0x009a71 (0x351d8 bytes) 000189 1786940159.122915 mapwrite 0x02da43 thru 0x038396 (0xa954 bytes) 000190 1786940159.147850 trunc from 0x038397 to 0x013201 (0x25195 bytes) 000192 1786940159.152302 read 0x007acf thru 0x009cc2 (0x21f4 bytes) 000193 1786940159.152813 mapread 0x00bcf6 thru 0x0109c0 (0x4ccb bytes) 000196 1786940159.156841 trunc from 0x013202 to 0x03153d (0x1e33c bytes) 000199 1786940159.162037 read 0x022eaf thru 0x030198 (0xd2ea bytes) 000200 1786940159.165398 punch from 0x00af87 to 0x00e093 (0x310d bytes) 000201 1786940159.166972 read 0x02328a thru 0x02e968 (0xb6df bytes) 000202 1786940159.167435 mapwrite 0x02785a thru 0x032dd7 (0xb57e bytes) 000203 1786940159.179896 mapread 0x013228 thru 0x01d5ca (0xa3a3 bytes) 000204 1786940159.180953 fallocate 0x00c127 thru 0x015468 (0x9342 bytes) 000205 1786940159.182739 mapread 0x02e430 thru 0x02f28a (0xe5b bytes) 000206 1786940159.184765 MIRROR_EXTEND to 2 mirrors 000207 1786940159.356229 mapread 0x026737 thru 0x02f11b (0x89e5 bytes) 000208 1786940159.361838 fallocate 0x019f30 thru 0x01fbe2 (0x5cb3 bytes) 000209 1786940159.373805 MIRROR_RESYNC 2 mirrors 000210 1786940159.594309 mapwrite 0x004d16 thru 0x00c023 (0x730e bytes) 000211 1786940159.627485 punch from 0x024ab2 to 0x02fc7a (0xb1c9 bytes) 000212 1786940159.629736 punch from 0x01acbc to 0x01c3a0 (0x16e5 bytes) 000213 1786940159.631254 trunc from 0x01fc70 to 0x02cc31 (0xcfc2 bytes) 000214 1786940159.633007 trunc from 0x02cae4 to 0x032dd7 (0x62f4 bytes) 000215 1786940159.634606 trunc from 0x01ab89 to 0x0282e2 (0xd75a bytes) 000216 1786940159.636461 mapread 0x00d44c thru 0x013146 (0x5cfb bytes) 000218 1786940159.641332 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32dd8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32dd8): Mirror 2: 0xcf041965 (stale) CRC-32 checksum value for chunk [0, 0x32dd8): Mirror 1: 0xe304d106 Mirror 3: 0xe304d106 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32dd8: skipped 000219 1786940159.866654 trunc from 0x02199f to 0x02def1 (0xc553 bytes) 000220 1786940159.877343 MIRROR_RESYNC 3 mirrors 000221 1786940160.084793 read 0x023cd1 thru 0x026e7f (0x31af bytes) 000222 1786940160.086822 read 0x02fb06 thru 0x032dd7 (0x32d2 bytes) 000223 1786940160.089807 fallocate 0x02c1e9 thru 0x037f86 (0xbd9e bytes) 000224 1786940160.097584 trunc from 0x0020ca to 0x00bc59 (0x9b90 bytes) 000226 1786940160.099440 trunc from 0x014a01 to 0x018597 (0x3b97 bytes) 000228 1786940160.101175 mapwrite 0x014867 thru 0x01e27f (0x9a19 bytes) 000230 1786940160.122716 trunc from 0x00151d to 0x00f736 (0xe21a bytes) 000231 1786940160.126084 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x32dd8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32dd8): Mirror 2: 0xe304d106 (stale) Mirror 3: 0xe304d106 (stale) CRC-32 checksum value for chunk [0, 0x32dd8): Mirror 1: 0x5c2dcc0f Mirror 4: 0x5c2dcc0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32dd8: skipped 000232 1786940160.338016 mapwrite 0x01b303 thru 0x01db0f (0x280d bytes) 000233 1786940160.371968 mapread 0x00d7fc thru 0x010f81 (0x3786 bytes) 000236 1786940160.376199 trunc from 0x032dd8 to 0x022281 (0x10b56 bytes) 000237 1786940160.387306 mapread 0x0072e4 thru 0x0135b4 (0xc2d1 bytes) 000238 1786940160.388144 fallocate 0x0139e1 thru 0x019e5b (0x647b bytes) 000239 1786940160.390037 write 0x005deb thru 0x014331 (0xe547 bytes) 000240 1786940160.394099 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22282): Mirror 2: 0xeee54deb (stale) Mirror 4: 0x4363c7b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22282): Mirror 2: 0xeee54deb (stale) Mirror 4: 0x4363c7b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22282: skipped 000241 1786940160.469103 trunc from 0x01f2f1 to 0x022281 (0x2f91 bytes) 000243 1786940160.472077 MIRROR_RESYNC 3 mirrors 000244 1786940160.640983 read 0x015ae8 thru 0x020159 (0xa672 bytes) 000245 1786940160.643227 mapwrite 0x00a651 thru 0x011e31 (0x77e1 bytes) 000246 1786940160.674179 read 0x00ff5e thru 0x0104fc (0x59f bytes) 000247 1786940160.676099 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x22282): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22282): Mirror 2: 0x979437c8 (stale) Mirror 4: 0x979437c8 (stale) CRC-32 checksum value for chunk [0, 0x22282): Mirror 1: 0xb4afac12 Mirror 5: 0xb4afac12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22282: skipped 000248 1786940160.883265 write 0x03c5d6 thru 0x03ffff (0x3a2a bytes) 000249 1786940160.893760 read 0x015d3a thru 0x020a51 (0xad18 bytes) 000250 1786940160.896392 mapwrite 0x028f85 thru 0x02a002 (0x107e bytes) 000251 1786940160.905933 mapread 0x025363 thru 0x02c319 (0x6fb7 bytes) 000252 1786940160.907107 mapread 0x00abfd thru 0x01816e (0xd572 bytes) 000253 1786940160.911412 fallocate 0x00298c thru 0x003ada (0x114f bytes) 000254 1786940160.913195 trunc from 0x007149 to 0x00960c (0x24c4 bytes) 000255 1786940160.914818 write 0x00cc23 thru 0x01693a (0x9d18 bytes) 000256 1786940160.915616 trunc from 0x02b392 to 0x039ff1 (0xec60 bytes) 000257 1786940160.917220 read 0x02c925 thru 0x02ee29 (0x2505 bytes) 000258 1786940160.919858 mapread 0x0171c9 thru 0x01d613 (0x644b bytes) 000259 1786940160.920556 trunc from 0x040000 to 0x02fc08 (0x103f7 bytes) 000260 1786940160.929676 mapread 0x02d2e0 thru 0x02fc08 (0x2929 bytes) 000261 1786940160.930352 mapwrite 0x019c85 thru 0x01b56e (0x18ea bytes) 000262 1786940160.934826 mapwrite 0x03d755 thru 0x03ffff (0x28ab bytes) 000264 1786940160.946114 mapread 0x00a5b2 thru 0x00eb8b (0x45da bytes) 000265 1786940160.946769 trunc from 0x040000 to 0x0297ad (0x16852 bytes) 000266 1786940160.950761 mapwrite 0x03b86b thru 0x03ffff (0x4795 bytes) 000267 1786940160.965405 punch from 0x0201ef to 0x02bcd7 (0xbae9 bytes) 000268 1786940160.967382 fallocate 0x0389c3 thru 0x03ffff (0x763d bytes) 000269 1786940160.969293 mapread 0x038995 thru 0x03e16c (0x57d8 bytes) 000270 1786940160.974092 fallocate 0x03cfad thru 0x03ffff (0x3053 bytes) 000271 1786940160.975999 mapread 0x013f55 thru 0x01ceef (0x8f9b bytes) 000272 1786940160.976784 read 0x031971 thru 0x03b65e (0x9cee bytes) 000273 1786940160.980083 trunc from 0x01f186 to 0x020112 (0xf8d bytes) 000274 1786940160.981816 punch from 0x03b3a8 to 0x03e3c3 (0x301c bytes) 000275 1786940160.983704 punch from 0x02e96e to 0x034ddc (0x646f bytes) 000276 1786940160.985525 punch from 0x02ac29 to 0x02b5a9 (0x981 bytes) 000277 1786940160.987252 write 0x01030b thru 0x011a50 (0x1746 bytes) 000278 1786940160.987945 mapread 0x024556 thru 0x02503a (0xae5 bytes) 000279 1786940160.988662 trunc from 0x040000 to 0x01eec4 (0x2113b bytes) 000280 1786940160.993529 write 0x0030c4 thru 0x00a0ac (0x6fe9 bytes) 000281 1786940160.996055 MIRROR_RESYNC 4 mirrors 000282 1786940161.193784 punch from 0x000e81 to 0x00dcbe (0xce3e bytes) 000283 1786940161.202764 read 0x01bf7d thru 0x01eec4 (0x2f48 bytes) 000284 1786940161.205023 mapwrite 0x01f758 thru 0x02adee (0xb697 bytes) 000286 1786940161.224245 write 0x03399e thru 0x034bb9 (0x121c bytes) 000287 1786940161.226455 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34bba): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34bba): Mirror 2: 0x4d3924be (stale) Mirror 4: 0x4d3924be (stale) Mirror 5: 0x4d3924be (stale) CRC-32 checksum value for chunk [0, 0x34bba): Mirror 1: 0x7a24cd45 Mirror 6: 0x7a24cd45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34bba: skipped 000288 1786940161.446329 trunc from 0x034bba to 0x00b9f9 (0x291c0 bytes) 000289 1786940161.464987 mapread 0x0096ed thru 0x00b9f9 (0x230d bytes) 000290 1786940161.470640 write 0x0382d3 thru 0x03cdb2 (0x4ae0 bytes) 000291 1786940161.471900 trunc from 0x03cdb3 to 0x0205b1 (0x1c801 bytes) 000292 1786940161.478174 punch from 0x01f1c2 to 0x0205b1 (0x13f0 bytes) 000293 1786940161.479976 mapwrite 0x011313 thru 0x01d3ad (0xc09b bytes) 000294 1786940161.492113 write 0x0281d5 thru 0x037aa0 (0xf8cc bytes) 000295 1786940161.494735 MIRROR_RESYNC 5 mirrors 000296 1786940161.762815 trunc from 0x037aa1 to 0x011f7f (0x25b21 bytes) 000297 1786940161.776716 mapread 0x004860 thru 0x00fdf9 (0xb59a bytes) 000298 1786940161.781559 punch from 0x00d402 to 0x011f7f (0x4b7e bytes) 000299 1786940161.783252 punch from 0x00f1a7 to 0x011f7f (0x2dd9 bytes) 000301 1786940161.785236 write 0x025ddb thru 0x02e253 (0x8479 bytes) 000302 1786940161.786129 punch from 0x02625c to 0x02a687 (0x442c bytes) 000303 1786940161.789807 mapwrite 0x03207d thru 0x03c5d1 (0xa555 bytes) 000304 1786940161.805118 fallocate 0x0007cc thru 0x00c2b0 (0xbae5 bytes) 000306 1786940161.807581 trunc from 0x00a57e to 0x0117da (0x725d bytes) 000307 1786940161.809397 write 0x03e0ab thru 0x03ffff (0x1f55 bytes) 000308 1786940161.810307 fallocate 0x01e177 thru 0x02c59f (0xe429 bytes) 000309 1786940161.812238 write 0x012c68 thru 0x019b8a (0x6f23 bytes) 000310 1786940161.815758 trunc from 0x040000 to 0x03dfaf (0x2050 bytes) 000311 1786940161.820634 trunc from 0x03dfb0 to 0x00b680 (0x3292f bytes) 000312 1786940161.825200 mapread 0x002c1c thru 0x00b680 (0x8a65 bytes) 000314 1786940161.827894 read 0x0013c2 thru 0x00b680 (0xa2bf bytes) 000315 1786940161.828439 fallocate 0x03e324 thru 0x03ffff (0x1cdc bytes) 000316 1786940161.830807 trunc from 0x040000 to 0x0313da (0xec25 bytes) 000317 1786940161.835525 trunc from 0x018e5d to 0x021619 (0x87bd bytes) 000318 1786940161.838945 MIRROR_RESYNC 5 mirrors 000320 1786940162.035170 trunc from 0x0313db to 0x020d9a (0x10640 bytes) 000322 1786940162.050259 mapwrite 0x036435 thru 0x03b67a (0x5246 bytes) 000324 1786940162.065263 punch from 0x024a21 to 0x024c34 (0x214 bytes) 000325 1786940162.066963 mapread 0x005fc1 thru 0x0132a6 (0xd2e6 bytes) 000326 1786940162.072717 mapwrite 0x02bf64 thru 0x02e57a (0x2617 bytes) 000327 1786940162.080265 mapwrite 0x01fed7 thru 0x02610b (0x6235 bytes) 000328 1786940162.087947 trunc from 0x03b67b to 0x03a411 (0x1269 bytes) 000329 1786940162.092555 read 0x0058aa thru 0x00ea99 (0x91f0 bytes) 000330 1786940162.093052 trunc from 0x03a412 to 0x02ab7d (0xf894 bytes) 000331 1786940162.097155 punch from 0x024d33 to 0x028eea (0x41b8 bytes) 000332 1786940162.099030 trunc from 0x02ab7e to 0x002eb4 (0x27cc9 bytes) 000333 1786940162.103093 fallocate 0x039508 thru 0x03c9c9 (0x34c2 bytes) 000334 1786940162.106349 MIRROR_RESYNC 5 mirrors 000335 1786940162.295384 mapwrite 0x03e215 thru 0x03ffff (0x1deb bytes) 000336 1786940162.312923 mapread 0x0121ee thru 0x018fed (0x6e00 bytes) 000337 1786940162.316193 fallocate 0x00a081 thru 0x00b30c (0x128c bytes) 000338 1786940162.318335 punch from 0x019f2a to 0x01b5a6 (0x167d bytes) 000339 1786940162.320453 trunc from 0x040000 to 0x03aa4d (0x55b2 bytes) 000340 1786940162.325972 write 0x037a22 thru 0x03ffff (0x85de bytes) 000341 1786940162.328556 punch from 0x0329e2 to 0x03a24d (0x786c bytes) 000343 1786940162.333547 punch from 0x014652 to 0x021afd (0xd4ac bytes) 000344 1786940162.335219 punch from 0x019433 to 0x0216d2 (0x82a0 bytes) 000345 1786940162.336529 trunc from 0x0008a4 to 0x0061ef (0x594c bytes) 000346 1786940162.338130 trunc from 0x040000 to 0x02e963 (0x1169c bytes) 000348 1786940162.342741 read 0x005fb9 thru 0x00f46e (0x94b6 bytes) 000349 1786940162.346024 mapwrite 0x0385ed thru 0x03f68b (0x709f bytes) 000350 1786940162.357909 trunc from 0x03f68c to 0x01dabe (0x21bcd bytes) 000351 1786940162.362328 trunc from 0x01d639 to 0x01dabe (0x486 bytes) 000352 1786940162.363795 write 0x02917a thru 0x032650 (0x94d7 bytes) 000353 1786940162.364929 trunc from 0x0209e0 to 0x027e7e (0x749f bytes) 000354 1786940162.366566 punch from 0x01ee05 to 0x0234ad (0x46a9 bytes) 000355 1786940162.367956 mapread 0x019fd2 thru 0x020edb (0x6f0a bytes) 000356 1786940162.370992 trunc from 0x032651 to 0x022c56 (0xf9fa bytes) 000357 1786940162.375591 write 0x01076f thru 0x017bb4 (0x7446 bytes) 000358 1786940162.376510 write 0x0296b5 thru 0x02ade9 (0x1735 bytes) 000359 1786940162.377155 punch from 0x02710f to 0x02ade9 (0x3cdb bytes) 000360 1786940162.382320 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2adea): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2adea): Mirror 2: 0xde9c3e96 (stale) Mirror 4: 0xde9c3e96 (stale) Mirror 5: 0xde9c3e96 (stale) Mirror 6: 0xde9c3e96 (stale) CRC-32 checksum value for chunk [0, 0x2adea): Mirror 1: 0xf4903f72 Mirror 7: 0xf4903f72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2adea: skipped 000361 1786940162.620138 mapwrite 0x020565 thru 0x02f755 (0xf1f1 bytes) 000362 1786940162.652774 trunc from 0x00a23a to 0x015223 (0xafea bytes) 000363 1786940162.654557 trunc from 0x0251e4 to 0x02c900 (0x771d bytes) 000364 1786940162.656217 trunc from 0x02f756 to 0x00f99f (0x1fdb6 bytes) 000365 1786940162.661572 punch from 0x005e3a to 0x00d47c (0x7643 bytes) 000366 1786940162.663315 write 0x0211ee thru 0x02e6b7 (0xd4ca bytes) 000367 1786940162.664475 punch from 0x00660f to 0x006a10 (0x402 bytes) 000368 1786940162.666006 trunc from 0x029ea1 to 0x02ab57 (0xcb7 bytes) 000369 1786940162.670047 read 0x01d9bb thru 0x022966 (0x4fac bytes) 000370 1786940162.671938 mapread 0x0211f8 thru 0x02e6b7 (0xd4c0 bytes) 000371 1786940162.674060 read 0x00b7f6 thru 0x00cf7c (0x1787 bytes) 000372 1786940162.677085 mapread 0x009eda thru 0x0155b2 (0xb6d9 bytes) 000373 1786940162.678269 mapwrite 0x003d82 thru 0x009710 (0x598f bytes) 000374 1786940162.684509 mapwrite 0x007fbf thru 0x0099c2 (0x1a04 bytes) 000375 1786940162.688338 trunc from 0x02e6b8 to 0x035d67 (0x76b0 bytes) 000376 1786940162.692146 punch from 0x012189 to 0x01b6fc (0x9574 bytes) 000377 1786940162.693617 mapread 0x01dbbb thru 0x026c5c (0x90a2 bytes) 000378 1786940162.694329 write 0x00ab16 thru 0x00db94 (0x307f bytes) 000379 1786940162.694997 trunc from 0x0096d9 to 0x013649 (0x9f71 bytes) 000380 1786940162.699745 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35d68): Mirror 2: 0x85c9f028 (stale) Mirror 4: 0x85c9f028 (stale) Mirror 5: 0x85c9f028 (stale) Mirror 6: 0x85c9f028 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35d68): Mirror 2: 0x85c9f028 (stale) Mirror 4: 0x85c9f028 (stale) Mirror 5: 0x85c9f028 (stale) Mirror 6: 0x85c9f028 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35d68: skipped 000381 1786940162.799109 mapwrite 0x02ab94 thru 0x03428f (0x96fc bytes) 000382 1786940162.813983 read 0x00dff7 thru 0x016686 (0x8690 bytes) 000383 1786940162.817872 punch from 0x02e56f to 0x035d67 (0x77f9 bytes) 000385 1786940162.820681 fallocate 0x01a8f1 thru 0x0249e0 (0xa0f0 bytes) 000386 1786940162.822931 read 0x00e792 thru 0x011253 (0x2ac2 bytes) 000388 1786940162.823724 mapwrite 0x028d09 thru 0x03817b (0xf473 bytes) 000389 1786940162.842837 trunc from 0x03817c to 0x01d4f6 (0x1ac85 bytes) 000390 1786940162.848865 trunc from 0x016540 to 0x01d4f6 (0x6fb7 bytes) 000391 1786940162.854334 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d4f7): Mirror 2: 0x4619d2d6 (stale) Mirror 4: 0x4619d2d6 (stale) Mirror 5: 0x4619d2d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d4f7): Mirror 2: 0x4619d2d6 (stale) Mirror 4: 0x4619d2d6 (stale) Mirror 5: 0x4619d2d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d4f7: skipped 000392 1786940162.961572 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d4f7): Mirror 4: 0x4619d2d6 (stale) Mirror 5: 0x4619d2d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d4f7): Mirror 4: 0x4619d2d6 (stale) Mirror 5: 0x4619d2d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d4f7: skipped 000393 1786940163.053675 fallocate 0x021269 thru 0x030eaf (0xfc47 bytes) 000395 1786940163.056198 mapread 0x00e018 thru 0x01c445 (0xe42e bytes) 000396 1786940163.064350 mapread 0x014833 thru 0x01d4f6 (0x8cc4 bytes) 000397 1786940163.065250 trunc from 0x01d4f7 to 0x01b7b1 (0x1d45 bytes) 000398 1786940163.071065 fallocate 0x01cc60 thru 0x027134 (0xa4d5 bytes) 000399 1786940163.072819 trunc from 0x027135 to 0x019824 (0xd910 bytes) 000400 1786940163.078564 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19825): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19825): Mirror 4: 0x1686fd83 (stale) Mirror 5: 0x1686fd83 (stale) CRC-32 checksum value for chunk [0, 0x19825): Mirror 1: 0x9af839f4 Mirror 6: 0x9af839f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19825: skipped 000402 1786940163.299307 mapread 0x0056d2 thru 0x012662 (0xcf91 bytes) 000403 1786940163.309321 punch from 0x000de2 to 0x005921 (0x4b40 bytes) 000404 1786940163.319113 mapwrite 0x0103e5 thru 0x01187b (0x1497 bytes) 000405 1786940163.326167 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19825): Mirror 5: 0x1686fd83 (stale) Mirror 6: 0x9af839f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19825): Mirror 5: 0x1686fd83 (stale) Mirror 6: 0x9af839f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19825: skipped 000406 1786940163.407844 fallocate 0x03d5b2 thru 0x03ffff (0x2a4e bytes) 000407 1786940163.410411 punch from 0x0093d8 to 0x014d12 (0xb93b bytes) 000409 1786940163.413388 trunc from 0x0178a9 to 0x02486e (0xcfc6 bytes) 000410 1786940163.415595 trunc from 0x040000 to 0x0225dc (0x1da23 bytes) 000411 1786940163.421091 trunc from 0x01df0b to 0x0225dc (0x46d2 bytes) 000413 1786940163.423313 punch from 0x00a98c to 0x00d2ab (0x2920 bytes) 000414 1786940163.425057 write 0x009109 thru 0x00f0df (0x5fd7 bytes) 000415 1786940163.428075 fallocate 0x012495 thru 0x01f4b9 (0xd025 bytes) 000416 1786940163.429633 write 0x03e690 thru 0x03ffff (0x1970 bytes) 000417 1786940163.430427 trunc from 0x01643b to 0x01cc34 (0x67fa bytes) 000418 1786940163.431951 trunc from 0x029f39 to 0x03173f (0x7807 bytes) 000419 1786940163.433347 mapread 0x010c93 thru 0x019b4b (0x8eb9 bytes) 000420 1786940163.438547 punch from 0x033422 to 0x0370a0 (0x3c7f bytes) 000421 1786940163.439900 mapwrite 0x004728 thru 0x00b8af (0x7188 bytes) 000424 1786940163.448773 mapread 0x000767 thru 0x008e34 (0x86ce bytes) 000426 1786940163.453105 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000428 1786940163.527728 punch from 0x010f9c to 0x020d48 (0xfdad bytes) 000429 1786940163.529775 punch from 0x03b91f to 0x03ffff (0x46e1 bytes) 000430 1786940163.531526 fallocate 0x038ade thru 0x03f260 (0x6783 bytes) 000431 1786940163.533406 write 0x012933 thru 0x02132b (0xe9f9 bytes) 000432 1786940163.540225 MIRROR_RESYNC 2 mirrors 000433 1786940163.716799 fallocate 0x0000a2 thru 0x00b925 (0xb884 bytes) 000434 1786940163.726507 mapread 0x0070e4 thru 0x00fb13 (0x8a30 bytes) 000435 1786940163.731688 mapread 0x002595 thru 0x011f87 (0xf9f3 bytes) 000436 1786940163.732506 mapread 0x010b97 thru 0x01f39f (0xe809 bytes) 000437 1786940163.738971 MIRROR_RESYNC 2 mirrors 000438 1786940163.907429 read 0x03aef5 thru 0x03ffff (0x510b bytes) 000440 1786940163.910197 fallocate 0x026e3f thru 0x0356fb (0xe8bd bytes) 000441 1786940163.919558 trunc from 0x021962 to 0x02aa1e (0x90bd bytes) 000442 1786940163.921131 mapread 0x0058da thru 0x013f3f (0xe666 bytes) 000443 1786940163.927879 fallocate 0x021207 thru 0x028572 (0x736c bytes) 000444 1786940163.929659 trunc from 0x00457f to 0x00ed94 (0xa816 bytes) 000445 1786940163.931171 trunc from 0x0034d5 to 0x006e57 (0x3983 bytes) 000446 1786940163.932630 trunc from 0x040000 to 0x0049a2 (0x3b65d bytes) 000447 1786940163.941726 read 0x0016d0 thru 0x0049a2 (0x32d3 bytes) 000448 1786940163.944006 punch from 0x0017fa to 0x0049a2 (0x31a9 bytes) 000449 1786940163.945431 write 0x019fc8 thru 0x01f46f (0x54a8 bytes) 000450 1786940163.946665 trunc from 0x0167f0 to 0x01f46f (0x8c80 bytes) 000451 1786940163.950835 write 0x02ad20 thru 0x032c34 (0x7f15 bytes) 000452 1786940163.952115 write 0x00f913 thru 0x01f571 (0xfc5f bytes) 000453 1786940163.957646 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32c35): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c35): Mirror 5: 0x782a7f3 (stale) CRC-32 checksum value for chunk [0, 0x32c35): Mirror 1: 0x80cc2ead Mirror 6: 0x80cc2ead lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c35: skipped 000454 1786940164.198901 punch from 0x00421d to 0x0065f9 (0x23dd bytes) 000455 1786940164.209760 MIRROR_RESYNC 3 mirrors 000456 1786940164.408549 read 0x00bbdd thru 0x0116d6 (0x5afa bytes) 000457 1786940164.411245 write 0x01e37d thru 0x02c5fa (0xe27e bytes) 000458 1786940164.422560 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32c35): Mirror 6: 0x80cc2ead (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c35): Mirror 6: 0x80cc2ead (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c35: skipped 000459 1786940164.494010 fallocate 0x02a878 thru 0x037333 (0xcabc bytes) 000460 1786940164.495942 trunc from 0x032c35 to 0x031b47 (0x10ed bytes) 000461 1786940164.501155 write 0x038189 thru 0x03ffff (0x7e77 bytes) 000462 1786940164.502263 mapread 0x036731 thru 0x03aafb (0x43cb bytes) 000463 1786940164.505696 trunc from 0x02546d to 0x031a82 (0xc616 bytes) 000464 1786940164.507924 punch from 0x02db97 to 0x038d23 (0xb18d bytes) 000465 1786940164.512835 read 0x03295c thru 0x03ffff (0xd6a4 bytes) 000466 1786940164.517246 write 0x0087fd thru 0x0185ad (0xfdb1 bytes) 000467 1786940164.521466 mapwrite 0x036bf6 thru 0x03ffff (0x940a bytes) 000468 1786940164.529277 fallocate 0x009374 thru 0x011665 (0x82f2 bytes) 000469 1786940164.531248 punch from 0x03cc48 to 0x03ffff (0x33b8 bytes) 000470 1786940164.533124 read 0x021ebf thru 0x02ea9e (0xcbe0 bytes) 000471 1786940164.536803 trunc from 0x018c38 to 0x019e24 (0x11ed bytes) 000472 1786940164.539009 mapread 0x021e26 thru 0x026a9a (0x4c75 bytes) 000473 1786940164.539733 mapread 0x006c1e thru 0x00eb4e (0x7f31 bytes) 000474 1786940164.540503 trunc from 0x040000 to 0x021341 (0x1ecbe bytes) 000476 1786940164.545295 fallocate 0x03da26 thru 0x03ffff (0x25da bytes) 000477 1786940164.546968 mapwrite 0x005bcd thru 0x013a4a (0xde7e bytes) 000478 1786940164.553901 read 0x010c06 thru 0x019bfa (0x8ff5 bytes) 000479 1786940164.555841 mapwrite 0x03d68f thru 0x03ffff (0x2971 bytes) 000480 1786940164.567169 punch from 0x0396cd to 0x03ffff (0x6933 bytes) 000481 1786940164.571160 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb79fad9a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe23a927 Mirror 7: 0xe23a927 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000482 1786940164.779252 trunc from 0x031b14 to 0x03ffff (0xe4ec bytes) 000483 1786940164.788041 read 0x03ccb7 thru 0x03ffff (0x3349 bytes) 000484 1786940164.790096 mapwrite 0x00f9a6 thru 0x018331 (0x898c bytes) 000485 1786940164.800439 mapread 0x0390db thru 0x03ffff (0x6f25 bytes) 000487 1786940164.804780 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb79fad9a (stale) Mirror 7: 0xe23a927 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc9d6a62e Mirror 8: 0xc9d6a62e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000488 1786940165.015945 mapwrite 0x00096d thru 0x004e83 (0x4517 bytes) 000489 1786940165.056230 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb79fad9a (stale) Mirror 7: 0xe23a927 (stale) Mirror 8: 0xc9d6a62e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x66a9661e Mirror 9: 0x66a9661e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000490 1786940165.308089 punch from 0x001875 to 0x00978f (0x7f1b bytes) 000491 1786940165.322742 trunc from 0x00bed1 to 0x010b8e (0x4cbe bytes) 000492 1786940165.324530 punch from 0x01db2e to 0x0296aa (0xbb7d bytes) 000493 1786940165.326232 mapread 0x02209d thru 0x0240d7 (0x203b bytes) 000494 1786940165.332734 fallocate 0x01d927 thru 0x0205c7 (0x2ca1 bytes) 000495 1786940165.334995 fallocate 0x03a221 thru 0x03f5ec (0x53cc bytes) 000496 1786940165.337410 punch from 0x01c35f to 0x01e17a (0x1e1c bytes) 000497 1786940165.341210 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb79fad9a (stale) Mirror 7: 0xe23a927 (stale) Mirror 8: 0xc9d6a62e (stale) Mirror 9: 0x66a9661e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe6400fa8 Mirror 10: 0xe6400fa8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000498 1786940165.596564 write 0x00b184 thru 0x00f2e4 (0x4161 bytes) 000500 1786940165.612160 read 0x01c54c thru 0x024a40 (0x84f5 bytes) 000501 1786940165.615064 mapread 0x00c37e thru 0x00e27b (0x1efe bytes) 000502 1786940165.615905 punch from 0x013cea to 0x01c89b (0x8bb2 bytes) 000503 1786940165.618188 trunc from 0x040000 to 0x03be26 (0x41d9 bytes) 000504 1786940165.632312 punch from 0x038222 to 0x03be26 (0x3c05 bytes) 000505 1786940165.634641 trunc from 0x00ff07 to 0x0129d1 (0x2acb bytes) 000507 1786940165.642854 write 0x000a24 thru 0x00a0a0 (0x967d bytes) 000508 1786940165.648164 fallocate 0x03820d thru 0x03ffff (0x7df3 bytes) 000510 1786940165.651310 trunc from 0x02a3bf to 0x0388cf (0xe511 bytes) 000511 1786940165.653571 mapwrite 0x000a9d thru 0x00afc5 (0xa529 bytes) 000512 1786940165.661910 fallocate 0x005d0e thru 0x013de2 (0xe0d5 bytes) 000513 1786940165.664312 write 0x00311a thru 0x006c7e (0x3b65 bytes) 000514 1786940165.665412 mapread 0x02a910 thru 0x034379 (0x9a6a bytes) 000515 1786940165.672197 mapread 0x01a60b thru 0x023ebd (0x98b3 bytes) 000516 1786940165.675858 trunc from 0x040000 to 0x01e934 (0x216cb bytes) 000517 1786940165.680977 trunc from 0x01d043 to 0x01e934 (0x18f2 bytes) 000518 1786940165.683243 trunc from 0x00c941 to 0x0164cc (0x9b8c bytes) 000519 1786940165.685572 trunc from 0x017ae3 to 0x01e934 (0x6e52 bytes) 000520 1786940165.687825 write 0x01853b thru 0x01bea9 (0x396f bytes) 000521 1786940165.691284 fallocate 0x01d33f thru 0x01e53d (0x11ff bytes) 000522 1786940165.693143 mapwrite 0x02eb93 thru 0x03e913 (0xfd81 bytes) 000523 1786940165.710566 write 0x00264c thru 0x004bcb (0x2580 bytes) 000524 1786940165.711243 mapwrite 0x00796f thru 0x011e39 (0xa4cb bytes) 000525 1786940165.720201 punch from 0x02021d to 0x02632d (0x6111 bytes) 000526 1786940165.721981 punch from 0x002069 to 0x00a2d0 (0x8268 bytes) 000527 1786940165.723861 trunc from 0x01aea1 to 0x02171d (0x687d bytes) 000528 1786940165.725986 read 0x0220dc thru 0x02d2e9 (0xb20e bytes) 000529 1786940165.729602 mapwrite 0x002f8a thru 0x00e125 (0xb19c bytes) 000531 1786940165.741344 punch from 0x0163b8 to 0x017e89 (0x1ad2 bytes) 000532 1786940165.743545 mapread 0x02fec2 thru 0x0371a7 (0x72e6 bytes) 000533 1786940165.744567 trunc from 0x03e914 to 0x034e46 (0x9acd bytes) 000534 1786940165.749271 trunc from 0x034e47 to 0x003f86 (0x30ec0 bytes) 000535 1786940165.754509 write 0x003b57 thru 0x00a60c (0x6ab6 bytes) 000536 1786940165.755648 mapwrite 0x01c593 thru 0x02459c (0x800a bytes) 000537 1786940165.779204 trunc from 0x0164cb to 0x01ddd3 (0x7909 bytes) 000538 1786940165.782575 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2459d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2459d): Mirror 6: 0x50f3de5d (stale) Mirror 7: 0x7ef68f5d (stale) Mirror 8: 0xc509dbed (stale) Mirror 9: 0xb3dba219 (stale) Mirror 10: 0x69f08280 (stale) CRC-32 checksum value for chunk [0, 0x2459d): Mirror 1: 0xfbe172ca Mirror 11: 0xfbe172ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2459d: skipped 000539 1786940166.027372 fallocate 0x0181cb thru 0x02023f (0x8075 bytes) 000540 1786940166.037071 mapread 0x0144cd thru 0x01b683 (0x71b7 bytes) 000541 1786940166.042870 trunc from 0x02459d to 0x0269cb (0x242f bytes) 000542 1786940166.053283 mapread 0x0117b7 thru 0x01673e (0x4f88 bytes) 000543 1786940166.053966 punch from 0x017744 to 0x01d569 (0x5e26 bytes) 000544 1786940166.055669 trunc from 0x0269cc to 0x01ed7b (0x7c50 bytes) 000546 1786940166.060618 mapread 0x0002f3 thru 0x0046d9 (0x43e7 bytes) 000547 1786940166.064056 trunc from 0x0164f3 to 0x01ed7b (0x8889 bytes) 000548 1786940166.065868 fallocate 0x007234 thru 0x00b0e7 (0x3eb4 bytes) 000549 1786940166.068019 read 0x00c79c thru 0x014d4f (0x85b4 bytes) 000550 1786940166.070386 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ed7c): Mirror 6: 0x7fc40c5a (stale) Mirror 8: 0x5065f9fa (stale) Mirror 9: 0xefce6de5 (stale) Mirror 10: 0x9f3e06cd (stale) Mirror 11: 0x2423a07c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ed7c): Mirror 6: 0x7fc40c5a (stale) Mirror 8: 0x5065f9fa (stale) Mirror 9: 0xefce6de5 (stale) Mirror 10: 0x9f3e06cd (stale) Mirror 11: 0x2423a07c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ed7c: skipped 000551 1786940166.171550 mapwrite 0x005016 thru 0x0076e5 (0x26d0 bytes) 000552 1786940166.182288 trunc from 0x01ed7c to 0x0201ea (0x146f bytes) 000554 1786940166.188440 mapwrite 0x01c99d thru 0x01ede1 (0x2445 bytes) 000555 1786940166.197330 write 0x00e53c thru 0x0153c6 (0x6e8b bytes) 000556 1786940166.198398 fallocate 0x0154cd thru 0x017252 (0x1d86 bytes) 000557 1786940166.201011 write 0x036384 thru 0x0378b5 (0x1532 bytes) 000558 1786940166.202013 mapread 0x0271a6 thru 0x02ac46 (0x3aa1 bytes) 000559 1786940166.208837 write 0x01ac81 thru 0x02601d (0xb39d bytes) 000560 1786940166.210053 trunc from 0x0378b6 to 0x00462d (0x33288 bytes) 000561 1786940166.216171 read 0x000284 thru 0x00462d (0x43aa bytes) 000563 1786940166.218706 trunc from 0x00190b to 0x00462d (0x2d23 bytes) 000565 1786940166.221208 mapwrite 0x03930c thru 0x03d5bd (0x42b2 bytes) 000566 1786940166.235051 mapread 0x00b33e thru 0x01460a (0x92cd bytes) 000567 1786940166.240570 read 0x018122 thru 0x021dd8 (0x9cb7 bytes) 000569 1786940166.244093 punch from 0x0057a1 to 0x0099c6 (0x4226 bytes) 000571 1786940166.246513 mapwrite 0x00242e thru 0x011048 (0xec1b bytes) 000572 1786940166.260593 mapwrite 0x008783 thru 0x00d1ff (0x4a7d bytes) 000573 1786940166.266201 trunc from 0x02ac3e to 0x02c787 (0x1b4a bytes) 000574 1786940166.269528 MIRROR_RESYNC 6 mirrors 000575 1786940166.520621 fallocate 0x002480 thru 0x004793 (0x2314 bytes) 000576 1786940166.531642 trunc from 0x022ac9 to 0x028247 (0x577f bytes) 000577 1786940166.533882 mapread 0x02d54c thru 0x030503 (0x2fb8 bytes) 000578 1786940166.539891 mapwrite 0x018e00 thru 0x028437 (0xf638 bytes) 000579 1786940166.555131 mapwrite 0x017d1f thru 0x0206d1 (0x89b3 bytes) 000581 1786940166.563178 trunc from 0x03d5be to 0x02ff3c (0xd681 bytes) 000583 1786940166.574404 write 0x011edf thru 0x0163a6 (0x44c8 bytes) 000584 1786940166.577209 MIRROR_RESYNC 6 mirrors 000586 1786940166.824328 fallocate 0x02bc89 thru 0x03463e (0x89b6 bytes) 000587 1786940166.836636 mapwrite 0x030b66 thru 0x0323b2 (0x184d bytes) 000589 1786940166.846984 trunc from 0x03463f to 0x0113f0 (0x2324e bytes) 000591 1786940166.857926 punch from 0x00f9c2 to 0x0113f0 (0x1a2f bytes) 000592 1786940166.859696 write 0x005a24 thru 0x012d57 (0xd334 bytes) 000593 1786940166.862902 write 0x001abf thru 0x004947 (0x2e89 bytes) 000594 1786940166.866003 trunc from 0x012d58 to 0x022418 (0xf6c1 bytes) 000595 1786940166.870111 write 0x03249a thru 0x03ffff (0xdb66 bytes) 000596 1786940166.873137 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x224e78ca (stale) Mirror 8: 0x224e78ca (stale) Mirror 9: 0x224e78ca (stale) Mirror 10: 0x224e78ca (stale) Mirror 11: 0x224e78ca (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xba503675 Mirror 12: 0xba503675 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1786940167.107555 punch from 0x005f91 to 0x00d7ac (0x781c bytes) 000598 1786940167.120825 read 0x000c3d thru 0x007f14 (0x72d8 bytes) 000599 1786940167.126130 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x224e78ca (stale) Mirror 9: 0x224e78ca (stale) Mirror 10: 0x224e78ca (stale) Mirror 11: 0x224e78ca (stale) Mirror 12: 0xba503675 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x224e78ca (stale) Mirror 9: 0x224e78ca (stale) Mirror 10: 0x224e78ca (stale) Mirror 11: 0x224e78ca (stale) Mirror 12: 0xba503675 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000600 1786940167.243806 fallocate 0x029ff2 thru 0x03458c (0xa59b bytes) 000601 1786940167.245967 mapread 0x00b89f thru 0x014af4 (0x9256 bytes) 000602 1786940167.252696 write 0x023644 thru 0x026d59 (0x3716 bytes) 000603 1786940167.257149 mapread 0x02feaa thru 0x039c18 (0x9d6f bytes) 000604 1786940167.262766 punch from 0x03baa9 to 0x03ed15 (0x326d bytes) 000605 1786940167.264889 mapwrite 0x00ec75 thru 0x0132ed (0x4679 bytes) 000606 1786940167.273344 MIRROR_RESYNC 6 mirrors 000607 1786940167.673150 write 0x007c9a thru 0x011c87 (0x9fee bytes) 000608 1786940167.685925 mapread 0x02fca9 thru 0x03be84 (0xc1dc bytes) 000609 1786940167.695512 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xf5a1f5dc (stale) Mirror 10: 0xf5a1f5dc (stale) Mirror 11: 0xf5a1f5dc (stale) Mirror 12: 0xf5a1f5dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xf5a1f5dc (stale) Mirror 10: 0xf5a1f5dc (stale) Mirror 11: 0xf5a1f5dc (stale) Mirror 12: 0xf5a1f5dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000610 1786940167.818909 mapwrite 0x027f4d thru 0x02dfa7 (0x605b bytes) 000611 1786940167.831682 mapread 0x00bbce thru 0x01748a (0xb8bd bytes) 000612 1786940167.838551 fallocate 0x013beb thru 0x020083 (0xc499 bytes) 000613 1786940167.840741 trunc from 0x028df4 to 0x033594 (0xa7a1 bytes) 000614 1786940167.842830 mapwrite 0x03710c thru 0x03ffff (0x8ef4 bytes) 000615 1786940167.853186 mapwrite 0x03a53f thru 0x03a9e2 (0x4a4 bytes) 000616 1786940167.858259 write 0x036217 thru 0x03b47b (0x5265 bytes) 000617 1786940167.860622 read 0x032923 thru 0x039e04 (0x74e2 bytes) 000618 1786940167.861046 read 0x02bcc7 thru 0x033834 (0x7b6e bytes) 000619 1786940167.864322 punch from 0x00fd30 to 0x016fbe (0x728f bytes) 000620 1786940167.866560 trunc from 0x040000 to 0x036b8c (0x9473 bytes) 000621 1786940167.877877 trunc from 0x036b8d to 0x024be8 (0x11fa4 bytes) 000622 1786940167.883384 trunc from 0x024be9 to 0x003763 (0x21485 bytes) 000623 1786940167.887818 trunc from 0x003764 to 0x0222fa (0x1eb97 bytes) 000624 1786940167.891945 trunc from 0x0222fb to 0x033300 (0x11006 bytes) 000625 1786940167.896106 read 0x01232b thru 0x01f485 (0xd15b bytes) 000626 1786940167.899894 punch from 0x016ba1 to 0x016ef8 (0x358 bytes) 000629 1786940167.903084 trunc from 0x033301 to 0x026d97 (0xc569 bytes) 000630 1786940167.909823 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x26d98): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26d98): Mirror 9: 0xe90ed8a9 (stale) Mirror 10: 0xe90ed8a9 (stale) Mirror 11: 0xe90ed8a9 (stale) Mirror 12: 0xe90ed8a9 (stale) CRC-32 checksum value for chunk [0, 0x26d98): Mirror 1: 0xe44ed9d2 Mirror 13: 0xe44ed9d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26d98: skipped 000631 1786940168.170985 trunc from 0x026d98 to 0x02edbf (0x8028 bytes) 000632 1786940168.182942 read 0x014f95 thru 0x01eab1 (0x9b1d bytes) 000633 1786940168.185856 write 0x038cae thru 0x03ffff (0x7352 bytes) 000634 1786940168.186829 read 0x0009b7 thru 0x0035e9 (0x2c33 bytes) 000635 1786940168.190032 punch from 0x021e16 to 0x025b06 (0x3cf1 bytes) 000636 1786940168.191861 mapwrite 0x033f78 thru 0x03ff99 (0xc022 bytes) 000637 1786940168.199980 read 0x031606 thru 0x031a7b (0x476 bytes) 000638 1786940168.200666 fallocate 0x02b70e thru 0x034902 (0x91f5 bytes) 000639 1786940168.202859 trunc from 0x007979 to 0x00cda2 (0x542a bytes) 000640 1786940168.204712 write 0x03e66d thru 0x03f16e (0xb02 bytes) 000641 1786940168.206862 MIRROR_RESYNC 6 mirrors 000642 1786940168.471199 write 0x014f5e thru 0x01c3a0 (0x7443 bytes) 000643 1786940168.481959 punch from 0x018592 to 0x01d550 (0x4fbf bytes) 000644 1786940168.485884 trunc from 0x00af86 to 0x013831 (0x88ac bytes) 000645 1786940168.487186 fallocate 0x01bae8 thru 0x0223de (0x68f7 bytes) 000646 1786940168.488726 mapwrite 0x025d48 thru 0x033fcd (0xe286 bytes) 000647 1786940168.500776 trunc from 0x001640 to 0x009df2 (0x87b3 bytes) 000648 1786940168.503479 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xea28784b (stale) Mirror 11: 0xea28784b (stale) Mirror 12: 0xea28784b (stale) Mirror 13: 0xea28784b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xea28784b (stale) Mirror 11: 0xea28784b (stale) Mirror 12: 0xea28784b (stale) Mirror 13: 0xea28784b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000649 1786940168.618996 fallocate 0x002184 thru 0x007be8 (0x5a65 bytes) 000650 1786940168.622628 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xea28784b (stale) Mirror 11: 0xea28784b (stale) Mirror 12: 0xea28784b (stale) Mirror 13: 0xea28784b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7dd17467 Mirror 14: 0x7dd17467 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000651 1786940168.869616 mapwrite 0x029553 thru 0x0326b2 (0x9160 bytes) 000652 1786940168.907804 punch from 0x014196 to 0x0240b2 (0xff1d bytes) 000653 1786940168.911190 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xea28784b (stale) Mirror 12: 0xea28784b (stale) Mirror 13: 0xea28784b (stale) Mirror 14: 0x7dd17467 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xea28784b (stale) Mirror 12: 0xea28784b (stale) Mirror 13: 0xea28784b (stale) Mirror 14: 0x7dd17467 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000654 1786940169.006178 punch from 0x0186d2 to 0x01add8 (0x2707 bytes) 000655 1786940169.007845 punch from 0x012d50 to 0x01947b (0x672c bytes) 000656 1786940169.009197 fallocate 0x02786e thru 0x033bf7 (0xc38a bytes) 000657 1786940169.010911 mapwrite 0x0074d0 thru 0x00b12c (0x3c5d bytes) 000658 1786940169.022117 write 0x00f0ec thru 0x012e63 (0x3d78 bytes) 000659 1786940169.025075 trunc from 0x040000 to 0x00f7a6 (0x30859 bytes) 000660 1786940169.034933 MIRROR_RESYNC 5 mirrors 000661 1786940169.306576 punch from 0x00b2ff to 0x00f7a6 (0x44a8 bytes) 000662 1786940169.314299 punch from 0x0003ec to 0x00f2c6 (0xeedb bytes) 000663 1786940169.317483 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf7a7): Mirror 12: 0xc6023da8 (stale) Mirror 13: 0xc6023da8 (stale) Mirror 14: 0xc6023da8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf7a7): Mirror 12: 0xc6023da8 (stale) Mirror 13: 0xc6023da8 (stale) Mirror 14: 0xc6023da8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf7a7: skipped 000664 1786940169.398297 fallocate 0x02c4b1 thru 0x02ff01 (0x3a51 bytes) 000665 1786940169.400222 write 0x02700a thru 0x036783 (0xf77a bytes) 000666 1786940169.404024 punch from 0x02594b to 0x0289ff (0x30b5 bytes) 000667 1786940169.407867 read 0x02d3f1 thru 0x033b75 (0x6785 bytes) 000668 1786940169.408404 write 0x02a524 thru 0x035671 (0xb14e bytes) 000669 1786940169.409397 trunc from 0x036784 to 0x035e5d (0x926 bytes) 000670 1786940169.413379 punch from 0x028ea8 to 0x02c4e1 (0x363a bytes) 000671 1786940169.417188 write 0x004c4c thru 0x00f70c (0xaac1 bytes) 000672 1786940169.420275 fallocate 0x023c61 thru 0x027109 (0x34a9 bytes) 000673 1786940169.422571 mapread 0x010668 thru 0x0177ea (0x7183 bytes) 000677 1786940169.429891 mapwrite 0x02be4b thru 0x032673 (0x6829 bytes) 000679 1786940169.441716 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35e5e): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e5e): Mirror 12: 0xe2a934e5 (stale) Mirror 13: 0xe2a934e5 (stale) Mirror 14: 0xe2a934e5 (stale) CRC-32 checksum value for chunk [0, 0x35e5e): Mirror 1: 0x399a70f4 Mirror 15: 0x399a70f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e5e: skipped 000680 1786940169.705251 fallocate 0x03409b thru 0x03d78a (0x96f0 bytes) 000681 1786940169.716669 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d78b): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d78b): Mirror 12: 0x51373659 (stale) Mirror 13: 0x51373659 (stale) Mirror 14: 0x51373659 (stale) Mirror 15: 0xa470d774 (stale) CRC-32 checksum value for chunk [0, 0x3d78b): Mirror 1: 0xa470d774 Mirror 16: 0xa470d774 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d78b: skipped 000682 1786940169.987677 trunc from 0x000485 to 0x00fd1b (0xf897 bytes) 000684 1786940169.998970 trunc from 0x03d78b to 0x0289aa (0x14de0 bytes) 000685 1786940170.009434 read 0x014331 thru 0x0147ad (0x47d bytes) 000686 1786940170.011354 punch from 0x021aa5 to 0x0255c6 (0x3b22 bytes) 000688 1786940170.013799 trunc from 0x022798 to 0x022ae2 (0x34b bytes) 000690 1786940170.017781 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x289ab): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x289ab): Mirror 12: 0xd60aadd7 (stale) Mirror 13: 0xd60aadd7 (stale) Mirror 14: 0xd60aadd7 (stale) Mirror 15: 0xed01944b (stale) Mirror 16: 0xed01944b (stale) CRC-32 checksum value for chunk [0, 0x289ab): Mirror 1: 0x862463d2 Mirror 17: 0x862463d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x289ab: skipped 000692 1786940170.301012 punch from 0x017486 to 0x020d0f (0x988a bytes) 000693 1786940170.316230 mapread 0x003be4 thru 0x00accd (0x70ea bytes) 000694 1786940170.323203 read 0x003343 thru 0x0126c8 (0xf386 bytes) 000695 1786940170.325749 write 0x01ef2a thru 0x02d426 (0xe4fd bytes) 000696 1786940170.331783 trunc from 0x02d427 to 0x02fe8e (0x2a68 bytes) 000697 1786940170.345616 trunc from 0x027acf to 0x0299c7 (0x1ef9 bytes) 000698 1786940170.350746 fallocate 0x02fedc thru 0x03fb32 (0xfc57 bytes) 000699 1786940170.352726 write 0x01ecd4 thru 0x0254d9 (0x6806 bytes) 000700 1786940170.355341 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fb33): Mirror 12: 0xee884c58 (stale) Mirror 13: 0xee884c58 (stale) Mirror 14: 0xee884c58 (stale) Mirror 16: 0xbaa11eb7 (stale) Mirror 17: 0xd362ece7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb33): Mirror 12: 0xee884c58 (stale) Mirror 13: 0xee884c58 (stale) Mirror 14: 0xee884c58 (stale) Mirror 16: 0xbaa11eb7 (stale) Mirror 17: 0xd362ece7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb33: skipped 000701 1786940170.492822 fallocate 0x03b1ec thru 0x03ffff (0x4e14 bytes) 000702 1786940170.495028 punch from 0x013acf to 0x02062e (0xcb60 bytes) 000704 1786940170.497075 punch from 0x02abae to 0x0339c0 (0x8e13 bytes) 000705 1786940170.498650 fallocate 0x00d575 thru 0x0109e6 (0x3472 bytes) 000706 1786940170.500447 trunc from 0x01fb10 to 0x02ee51 (0xf342 bytes) 000707 1786940170.502063 trunc from 0x01fdf0 to 0x02e3b9 (0xe5ca bytes) 000708 1786940170.503755 read 0x01fc4f thru 0x02c5f7 (0xc9a9 bytes) 000709 1786940170.507033 fallocate 0x01ce87 thru 0x025a9d (0x8c17 bytes) 000710 1786940170.508937 trunc from 0x02c74f to 0x03288c (0x613e bytes) 000711 1786940170.510767 write 0x039a87 thru 0x03a9b7 (0xf31 bytes) 000712 1786940170.515175 punch from 0x025353 to 0x03498c (0xf63a bytes) 000713 1786940170.517076 fallocate 0x00f8a7 thru 0x0123a6 (0x2b00 bytes) 000714 1786940170.519062 mapwrite 0x0387a3 thru 0x03ffff (0x785d bytes) 000717 1786940170.532719 read 0x003bfe thru 0x0101b1 (0xc5b4 bytes) 000718 1786940170.536682 mapread 0x037c3f thru 0x03d64f (0x5a11 bytes) 000719 1786940170.537387 fallocate 0x005fed thru 0x014a0c (0xea20 bytes) 000720 1786940170.539592 trunc from 0x040000 to 0x036a20 (0x95df bytes) 000721 1786940170.544749 trunc from 0x036a21 to 0x01b05a (0x1b9c6 bytes) 000722 1786940170.549694 mapwrite 0x01922b thru 0x02455f (0xb335 bytes) 000723 1786940170.573350 MIRROR_RESYNC 6 mirrors 000724 1786940170.897539 read 0x022a04 thru 0x02455f (0x1b5c bytes) 000726 1786940170.899517 fallocate 0x03c993 thru 0x03ffff (0x366d bytes) 000727 1786940170.912203 mapwrite 0x02d19e thru 0x033e6e (0x6cd1 bytes) 000728 1786940170.923994 mapread 0x02d2f9 thru 0x03409a (0x6da2 bytes) 000729 1786940170.926053 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xf9268186 (stale) Mirror 13: 0xf9268186 (stale) Mirror 16: 0xf9268186 (stale) Mirror 17: 0xf9268186 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xf9268186 (stale) Mirror 13: 0xf9268186 (stale) Mirror 16: 0xf9268186 (stale) Mirror 17: 0xf9268186 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000731 1786940171.037083 punch from 0x0229c3 to 0x031b46 (0xf184 bytes) 000732 1786940171.039781 trunc from 0x040000 to 0x025771 (0x1a88e bytes) 000733 1786940171.054772 punch from 0x00cd66 to 0x00f174 (0x240f bytes) 000734 1786940171.058596 MIRROR_RESYNC 5 mirrors 000735 1786940171.369552 write 0x033afd thru 0x0386ca (0x4bce bytes) 000736 1786940171.377906 fallocate 0x01713b thru 0x0229a5 (0xb86b bytes) 000737 1786940171.379777 read 0x01adbe thru 0x01bc26 (0xe69 bytes) 000738 1786940171.381528 read 0x001645 thru 0x00b7f4 (0xa1b0 bytes) 000739 1786940171.384338 mapread 0x020d38 thru 0x02dfad (0xd276 bytes) 000740 1786940171.385180 trunc from 0x0386cb to 0x001a10 (0x36cba bytes) 000742 1786940171.391413 trunc from 0x0001cf to 0x001a10 (0x1842 bytes) 000743 1786940171.392950 trunc from 0x001a11 to 0x0000fc (0x1914 bytes) 000744 1786940171.396583 punch from 0x000078 to 0x0000fc (0x85 bytes) 000745 1786940171.397993 write 0x02966e thru 0x02b330 (0x1cc3 bytes) 000746 1786940171.399050 write 0x033cb1 thru 0x03ffff (0xc34f bytes) 000747 1786940171.400044 trunc from 0x040000 to 0x0213ca (0x1ec35 bytes) 000748 1786940171.404046 write 0x0162a2 thru 0x023242 (0xcfa1 bytes) 000749 1786940171.406214 trunc from 0x023243 to 0x014b9a (0xe6a8 bytes) 000750 1786940171.409853 read 0x010952 thru 0x012c84 (0x2333 bytes) 000751 1786940171.414045 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14b9b): Mirror 12: 0x59a8bd3c (stale) Mirror 13: 0x59a8bd3c (stale) Mirror 16: 0x59a8bd3c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14b9b): Mirror 12: 0x59a8bd3c (stale) Mirror 13: 0x59a8bd3c (stale) Mirror 16: 0x59a8bd3c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14b9b: skipped 000752 1786940171.503429 fallocate 0x03f150 thru 0x03ffff (0xeb0 bytes) 000753 1786940171.505736 read 0x001638 thru 0x006363 (0x4d2c bytes) 000754 1786940171.508995 trunc from 0x014b9b to 0x008ca2 (0xbef8 bytes) 000755 1786940171.514670 trunc from 0x002d8f to 0x008ca2 (0x5f14 bytes) 000756 1786940171.516967 read 0x0068b5 thru 0x008ca2 (0x23ee bytes) 000757 1786940171.520027 mapread 0x004575 thru 0x008ca2 (0x472e bytes) 000758 1786940171.520735 write 0x03f442 thru 0x03ffff (0xbbe bytes) 000759 1786940171.521590 trunc from 0x040000 to 0x029887 (0x16778 bytes) 000760 1786940171.526121 trunc from 0x029888 to 0x03e0c3 (0x1483c bytes) 000761 1786940171.529814 read 0x0185f1 thru 0x020f22 (0x8932 bytes) 000762 1786940171.533354 write 0x036c9b thru 0x03ffff (0x9365 bytes) 000763 1786940171.534836 read 0x01522a thru 0x01cf31 (0x7d08 bytes) 000764 1786940171.535537 fallocate 0x03ff18 thru 0x03ffff (0xe8 bytes) 000765 1786940171.537591 mapread 0x03aed6 thru 0x03ffff (0x512a bytes) 000767 1786940171.538755 punch from 0x020f88 to 0x02d044 (0xc0bd bytes) 000771 1786940171.541644 mapread 0x004a5b thru 0x005558 (0xafe bytes) 000772 1786940171.542254 trunc from 0x02648b to 0x02bf01 (0x5a77 bytes) 000773 1786940171.544404 mapwrite 0x027bb8 thru 0x02d2e6 (0x572f bytes) 000774 1786940171.554386 mapread 0x00818f thru 0x00a000 (0x1e72 bytes) 000775 1786940171.555109 write 0x00fe9f thru 0x011770 (0x18d2 bytes) 000776 1786940171.555806 trunc from 0x02b790 to 0x02c9d0 (0x1241 bytes) 000777 1786940171.557398 mapread 0x027317 thru 0x028e71 (0x1b5b bytes) 000778 1786940171.558238 trunc from 0x03ca43 to 0x03d040 (0x5fe bytes) 000779 1786940171.559982 mapread 0x0184d9 thru 0x026c8e (0xe7b6 bytes) 000780 1786940171.560782 read 0x02de58 thru 0x03be0f (0xdfb8 bytes) 000781 1786940171.561289 mapwrite 0x028cb6 thru 0x03467d (0xb9c8 bytes) 000782 1786940171.572310 trunc from 0x00be2c to 0x00c561 (0x736 bytes) 000783 1786940171.574090 read 0x0048b1 thru 0x013ce3 (0xf433 bytes) 000784 1786940171.576254 mapread 0x036223 thru 0x03ffff (0x9ddd bytes) 000785 1786940171.577319 trunc from 0x00907f to 0x00c3df (0x3361 bytes) 000786 1786940171.579333 trunc from 0x040000 to 0x02b995 (0x1466a bytes) 000788 1786940171.583898 mapread 0x0266a2 thru 0x028c56 (0x25b5 bytes) 000789 1786940171.584706 punch from 0x00f8c8 to 0x01e0a4 (0xe7dd bytes) 000791 1786940171.586956 write 0x010b0b thru 0x017b8f (0x7085 bytes) 000792 1786940171.591751 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2b996): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b996): Mirror 12: 0x7dde4ff (stale) Mirror 13: 0x7dde4ff (stale) Mirror 16: 0x7dde4ff (stale) CRC-32 checksum value for chunk [0, 0x2b996): Mirror 1: 0x619b4b22 Mirror 17: 0x619b4b22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b996: skipped 000793 1786940171.814576 mapwrite 0x017d94 thru 0x01f8dc (0x7b49 bytes) 000794 1786940171.855281 mapread 0x0050d9 thru 0x00c662 (0x758a bytes) 000795 1786940171.859150 punch from 0x01098c to 0x011fac (0x1621 bytes) 000796 1786940171.863201 MIRROR_RESYNC 5 mirrors 000797 1786940172.141461 fallocate 0x00ee6a thru 0x015318 (0x64af bytes) 000798 1786940172.151316 fallocate 0x024484 thru 0x027aca (0x3647 bytes) 000799 1786940172.153144 trunc from 0x02b996 to 0x003b35 (0x27e60 bytes) 000800 1786940172.159283 punch from 0x00339d to 0x003b35 (0x799 bytes) 000802 1786940172.161322 mapwrite 0x03257f thru 0x039ea1 (0x7923 bytes) 000803 1786940172.176343 fallocate 0x03fd36 thru 0x03ffff (0x2ca bytes) 000804 1786940172.178280 read 0x02f0f0 thru 0x035ea2 (0x6db3 bytes) 000805 1786940172.180437 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x5cd6707a (stale) Mirror 13: 0x5cd6707a (stale) Mirror 16: 0x5cd6707a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x5cd6707a (stale) Mirror 13: 0x5cd6707a (stale) Mirror 16: 0x5cd6707a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000806 1786940172.273566 trunc from 0x017e5b to 0x025703 (0xd8a9 bytes) 000808 1786940172.275984 mapwrite 0x03c490 thru 0x03ffff (0x3b70 bytes) 000809 1786940172.286982 read 0x02a565 thru 0x02b82c (0x12c8 bytes) 000810 1786940172.289174 mapwrite 0x0122f3 thru 0x01cc05 (0xa913 bytes) 000811 1786940172.299715 fallocate 0x01e7b0 thru 0x02e4ea (0xfd3b bytes) 000812 1786940172.301834 mapwrite 0x0330d0 thru 0x03853f (0x5470 bytes) 000813 1786940172.311133 fallocate 0x035bf9 thru 0x03ffff (0xa407 bytes) 000815 1786940172.313963 read 0x009708 thru 0x0188d6 (0xf1cf bytes) 000816 1786940172.317185 fallocate 0x038198 thru 0x03ffff (0x7e68 bytes) 000817 1786940172.318927 punch from 0x01a0ba to 0x01effd (0x4f44 bytes) 000818 1786940172.322024 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x5cd6707a (stale) Mirror 13: 0x5cd6707a (stale) Mirror 16: 0x5cd6707a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ec89027 Mirror 17: 0x2ec89027 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000819 1786940172.559223 trunc from 0x040000 to 0x0278cb (0x18734 bytes) 000820 1786940172.573999 mapread 0x0166d1 thru 0x025834 (0xf164 bytes) 000821 1786940172.581485 punch from 0x00e4da to 0x0180b6 (0x9bdd bytes) 000822 1786940172.583214 mapread 0x0049b2 thru 0x009f9f (0x55ee bytes) 000823 1786940172.586151 trunc from 0x0278cc to 0x03fe0f (0x18544 bytes) 000824 1786940172.591547 write 0x0322e7 thru 0x039c25 (0x793f bytes) 000826 1786940172.594882 write 0x01bcc3 thru 0x0226c9 (0x6a07 bytes) 000829 1786940172.596305 write 0x02de0e thru 0x036193 (0x8386 bytes) 000831 1786940172.599283 punch from 0x01fce5 to 0x027811 (0x7b2d bytes) 000832 1786940172.604332 mapwrite 0x00668e thru 0x00ce85 (0x67f8 bytes) 000833 1786940172.611542 mapread 0x033aed thru 0x03d355 (0x9869 bytes) 000834 1786940172.614767 fallocate 0x00542e thru 0x013ad2 (0xe6a5 bytes) 000835 1786940172.616917 trunc from 0x03fe10 to 0x00995e (0x364b1 bytes) 000836 1786940172.622000 write 0x022449 thru 0x02c3fc (0x9fb4 bytes) 000837 1786940172.624800 MIRROR_RESYNC 5 mirrors 000838 1786940172.895206 trunc from 0x02c3fd to 0x036ba4 (0xa7a8 bytes) 000839 1786940172.909518 mapread 0x01a111 thru 0x01ff16 (0x5e06 bytes) 000840 1786940172.915524 write 0x00853f thru 0x017ec2 (0xf984 bytes) 000841 1786940172.920774 punch from 0x013af1 to 0x018ddb (0x52eb bytes) 000842 1786940172.925510 mapread 0x0277fc thru 0x028eee (0x16f3 bytes) 000843 1786940172.929731 mapwrite 0x01faa0 thru 0x02198d (0x1eee bytes) 000844 1786940172.934968 mapwrite 0x0394ac thru 0x039a5b (0x5b0 bytes) 000845 1786940172.947009 read 0x032c16 thru 0x039a5b (0x6e46 bytes) 000846 1786940172.947553 trunc from 0x0267f0 to 0x02eeb6 (0x86c7 bytes) 000847 1786940172.951010 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39a5c): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a5c): Mirror 12: 0xa1867a69 (stale) Mirror 13: 0xa1867a69 (stale) Mirror 16: 0xa1867a69 (stale) Mirror 17: 0xa1867a69 (stale) CRC-32 checksum value for chunk [0, 0x39a5c): Mirror 1: 0x2dfec471 Mirror 18: 0x2dfec471 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a5c: skipped 000848 1786940173.218214 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x39a5c): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a5c): Mirror 12: 0xa1867a69 (stale) Mirror 13: 0xa1867a69 (stale) Mirror 16: 0xa1867a69 (stale) Mirror 17: 0xa1867a69 (stale) CRC-32 checksum value for chunk [0, 0x39a5c): Mirror 1: 0x2dfec471 Mirror 18: 0x2dfec471 Mirror 19: 0x2dfec471 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a5c: skipped 000849 1786940173.496099 read 0x01f70a thru 0x027918 (0x820f bytes) 000850 1786940173.498713 trunc from 0x039a5c to 0x010dc4 (0x28c97 bytes) 000851 1786940173.511961 mapread 0x001c4d thru 0x0102ed (0xe6a1 bytes) 000852 1786940173.518012 punch from 0x00f6fc to 0x010dc4 (0x16c9 bytes) 000853 1786940173.519893 fallocate 0x009973 thru 0x014d87 (0xb415 bytes) 000854 1786940173.521655 punch from 0x000a29 to 0x00b16c (0xa744 bytes) 000855 1786940173.524631 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10dc5): Mirror 13: 0x408a08f6 (stale) Mirror 16: 0x408a08f6 (stale) Mirror 17: 0x408a08f6 (stale) Mirror 18: 0xfe40b306 (stale) Mirror 19: 0xfe40b306 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10dc5): Mirror 13: 0x408a08f6 (stale) Mirror 16: 0x408a08f6 (stale) Mirror 17: 0x408a08f6 (stale) Mirror 18: 0xfe40b306 (stale) Mirror 19: 0xfe40b306 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10dc5: skipped 000856 1786940173.618452 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10dc5): Mirror 13: 0x408a08f6 (stale) Mirror 16: 0x408a08f6 (stale) Mirror 17: 0x408a08f6 (stale) Mirror 18: 0xfe40b306 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10dc5): Mirror 13: 0x408a08f6 (stale) Mirror 16: 0x408a08f6 (stale) Mirror 17: 0x408a08f6 (stale) Mirror 18: 0xfe40b306 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10dc5: skipped 000857 1786940173.713267 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10dc5): Mirror 13: 0x408a08f6 (stale) Mirror 16: 0x408a08f6 (stale) Mirror 17: 0x408a08f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10dc5): Mirror 13: 0x408a08f6 (stale) Mirror 16: 0x408a08f6 (stale) Mirror 17: 0x408a08f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10dc5: skipped 000858 1786940173.794044 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10dc5): Mirror 13: 0x408a08f6 (stale) Mirror 17: 0x408a08f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10dc5): Mirror 13: 0x408a08f6 (stale) Mirror 17: 0x408a08f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10dc5: skipped 000859 1786940173.867912 punch from 0x00ed6d to 0x0101e6 (0x147a bytes) 000860 1786940173.869668 trunc from 0x008a1b to 0x00c9ab (0x3f91 bytes) 000861 1786940173.871056 trunc from 0x010dc5 to 0x009895 (0x752f bytes) 000862 1786940173.876095 mapwrite 0x00c759 thru 0x0162fc (0x9ba4 bytes) 000863 1786940173.891422 write 0x00bb1d thru 0x01a9f1 (0xeed5 bytes) 000864 1786940173.892519 trunc from 0x00c29b to 0x0187ba (0xc520 bytes) 000865 1786940173.896695 trunc from 0x00d972 to 0x013f69 (0x65f8 bytes) 000866 1786940173.898158 write 0x018d47 thru 0x02151b (0x87d5 bytes) 000867 1786940173.900643 read 0x01a4fb thru 0x0203d4 (0x5eda bytes) 000868 1786940173.901246 fallocate 0x02eb0d thru 0x0366b3 (0x7ba7 bytes) 000869 1786940173.903071 trunc from 0x033efe to 0x0366b3 (0x27b6 bytes) 000870 1786940173.904741 read 0x02d858 thru 0x0366b3 (0x8e5c bytes) 000871 1786940173.908243 trunc from 0x0366b4 to 0x01f2fe (0x173b5 bytes) 000872 1786940173.912824 trunc from 0x01f2ff to 0x03c0d0 (0x1cdd2 bytes) 000874 1786940173.918071 trunc from 0x007cb0 to 0x01565c (0xd9ad bytes) 000875 1786940173.919924 write 0x024d4a thru 0x02b7ba (0x6a71 bytes) 000876 1786940173.925014 MIRROR_RESYNC 3 mirrors 000879 1786940174.139899 fallocate 0x0058b8 thru 0x00bb41 (0x628a bytes) 000880 1786940174.147354 mapwrite 0x00a698 thru 0x00cb1e (0x2487 bytes) 000881 1786940174.156811 punch from 0x010d75 to 0x011444 (0x6d0 bytes) 000882 1786940174.158326 mapwrite 0x038f16 thru 0x03ffff (0x70ea bytes) 000883 1786940174.170608 fallocate 0x00e492 thru 0x01a90b (0xc47a bytes) 000885 1786940174.172846 fallocate 0x00953d thru 0x00b897 (0x235b bytes) 000887 1786940174.175445 mapread 0x0290ed thru 0x038afe (0xfa12 bytes) 000888 1786940174.180344 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x51ca9e15 (stale) Mirror 17: 0x51ca9e15 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xebebaa15 Mirror 18: 0xebebaa15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000889 1786940174.410217 trunc from 0x0193f3 to 0x020bdd (0x77eb bytes) 000890 1786940174.419715 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x51ca9e15 (stale) Mirror 18: 0xebebaa15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x51ca9e15 (stale) Mirror 18: 0xebebaa15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000891 1786940174.496775 mapwrite 0x004560 thru 0x00f5c0 (0xb061 bytes) 000892 1786940174.507568 read 0x017ad3 thru 0x0208bc (0x8dea bytes) 000893 1786940174.509827 mapread 0x028a12 thru 0x032409 (0x99f8 bytes) 000895 1786940174.514404 mapread 0x02cebd thru 0x0384d8 (0xb61c bytes) 000896 1786940174.515089 trunc from 0x002947 to 0x00b71b (0x8dd5 bytes) 000898 1786940174.516968 read 0x003780 thru 0x01071b (0xcf9c bytes) 000899 1786940174.519548 trunc from 0x001f0e to 0x00592b (0x3a1e bytes) 000900 1786940174.520988 fallocate 0x024928 thru 0x02a208 (0x58e1 bytes) 000901 1786940174.522649 mapwrite 0x024311 thru 0x02742c (0x311c bytes) 000902 1786940174.526894 read 0x03c52e thru 0x03e170 (0x1c43 bytes) 000903 1786940174.528516 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xebebaa15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xebebaa15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000904 1786940174.610813 punch from 0x022f28 to 0x02d30e (0xa3e7 bytes) 000905 1786940174.612948 read 0x004269 thru 0x009c0e (0x59a6 bytes) 000906 1786940174.617677 MIRROR_RESYNC 2 mirrors 000907 1786940174.788403 read 0x02992e thru 0x0398e4 (0xffb7 bytes) 000908 1786940174.790910 read 0x01475a thru 0x022f2b (0xe7d2 bytes) 000910 1786940174.793562 write 0x016a04 thru 0x0255d7 (0xebd4 bytes) 000911 1786940174.804164 write 0x039a80 thru 0x03d235 (0x37b6 bytes) 000912 1786940174.806914 mapwrite 0x0292c9 thru 0x02b6f6 (0x242e bytes) 000913 1786940174.816202 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xc32ed004 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfd26619c Mirror 19: 0xfd26619c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000915 1786940175.039162 mapread 0x02c2c2 thru 0x02eadc (0x281b bytes) 000916 1786940175.046991 read 0x02acb8 thru 0x02da7c (0x2dc5 bytes) 000918 1786940175.050344 trunc from 0x040000 to 0x01e27c (0x21d83 bytes) 000919 1786940175.065672 trunc from 0x01e27d to 0x02f0bd (0x10e41 bytes) 000920 1786940175.070904 fallocate 0x02924f thru 0x0356d3 (0xc485 bytes) 000921 1786940175.072459 write 0x01d680 thru 0x02b9ab (0xe32c bytes) 000922 1786940175.077195 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f0be): Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f0be): Mirror 18: 0xa29379f7 (stale) Mirror 19: 0xa5954be2 (stale) CRC-32 checksum value for chunk [0, 0x2f0be): Mirror 1: 0xc9642acf Mirror 20: 0xc9642acf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f0be: skipped 000923 1786940175.330121 mapread 0x003230 thru 0x00e833 (0xb604 bytes) 000924 1786940175.337663 read 0x00e4b2 thru 0x01bfdb (0xdb2a bytes) 000925 1786940175.341218 mapread 0x011a0a thru 0x013e03 (0x23fa bytes) 000926 1786940175.344125 mapwrite 0x0297fb thru 0x035807 (0xc00d bytes) 000927 1786940175.363961 mapread 0x003706 thru 0x004731 (0x102c bytes) 000928 1786940175.366493 trunc from 0x035808 to 0x02f6b9 (0x614e bytes) 000929 1786940175.371388 punch from 0x01b928 to 0x022e9b (0x7574 bytes) 000930 1786940175.373072 trunc from 0x02f6ba to 0x01c8a5 (0x12e14 bytes) 000931 1786940175.376592 write 0x03c595 thru 0x03ffff (0x3a6b bytes) 000932 1786940175.377332 mapwrite 0x01d3ea thru 0x022840 (0x5457 bytes) 000933 1786940175.384161 read 0x019dea thru 0x01c3c9 (0x25e0 bytes) 000934 1786940175.384582 punch from 0x004c1e to 0x00ac0a (0x5fed bytes) 000935 1786940175.385927 punch from 0x037d22 to 0x03ffff (0x82de bytes) 000936 1786940175.387280 mapread 0x032482 thru 0x03ea69 (0xc5e8 bytes) 000937 1786940175.390318 mapwrite 0x02e1aa thru 0x02e30a (0x161 bytes) 000939 1786940175.393915 write 0x02aeac thru 0x039f7e (0xf0d3 bytes) 000940 1786940175.394919 trunc from 0x040000 to 0x01f5aa (0x20a55 bytes) 000941 1786940175.399044 mapwrite 0x03cdb3 thru 0x03ffff (0x324d bytes) 000942 1786940175.410496 write 0x018f60 thru 0x01a358 (0x13f9 bytes) 000944 1786940175.411484 mapwrite 0x01b53f thru 0x028ea2 (0xd964 bytes) 000945 1786940175.430144 MIRROR_RESYNC 4 mirrors 000946 1786940175.675275 punch from 0x03c8eb to 0x03e9c8 (0x20de bytes) 000947 1786940175.685609 trunc from 0x040000 to 0x01dec4 (0x2213b bytes) 000948 1786940175.696220 mapwrite 0x03849c thru 0x03d362 (0x4ec7 bytes) 000949 1786940175.711542 read 0x03d143 thru 0x03d362 (0x220 bytes) 000950 1786940175.712007 punch from 0x003714 to 0x003baa (0x497 bytes) 000952 1786940175.713974 read 0x017336 thru 0x01c957 (0x5622 bytes) 000953 1786940175.718015 read 0x03b3e2 thru 0x03d362 (0x1f81 bytes) 000954 1786940175.718672 write 0x01d2d3 thru 0x01f7dc (0x250a bytes) 000956 1786940175.720146 read 0x018edf thru 0x01afc8 (0x20ea bytes) 000957 1786940175.720758 trunc from 0x004591 to 0x00f801 (0xb271 bytes) 000958 1786940175.723170 mapread 0x008cb5 thru 0x015579 (0xc8c5 bytes) 000960 1786940175.726808 mapread 0x01e05e thru 0x02769b (0x963e bytes) 000962 1786940175.729183 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d363): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d363): Mirror 18: 0xecff81e9 (stale) Mirror 19: 0xecff81e9 (stale) Mirror 20: 0xecff81e9 (stale) CRC-32 checksum value for chunk [0, 0x3d363): Mirror 1: 0x1a4ee482 Mirror 21: 0x1a4ee482 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d363: skipped 000963 1786940175.965163 trunc from 0x03d363 to 0x011603 (0x2bd5f bytes) 000964 1786940175.987335 trunc from 0x011346 to 0x011603 (0x2be bytes) 000966 1786940175.989400 mapwrite 0x03ac83 thru 0x03f78f (0x4b0d bytes) 000967 1786940176.005344 mapread 0x025b5a thru 0x026225 (0x6cc bytes) 000968 1786940176.008695 fallocate 0x028f71 thru 0x033e02 (0xae92 bytes) 000969 1786940176.011155 read 0x03805c thru 0x039d83 (0x1d28 bytes) 000970 1786940176.011823 trunc from 0x001521 to 0x00420d (0x2ced bytes) 000972 1786940176.013972 trunc from 0x03f790 to 0x019d72 (0x25a1d bytes) 000973 1786940176.018547 fallocate 0x0300c8 thru 0x03e921 (0xe85a bytes) 000974 1786940176.022697 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19d73): Mirror 19: 0xc40ad31e (stale) Mirror 20: 0xc40ad31e (stale) Mirror 21: 0xd922b1f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19d73): Mirror 19: 0xc40ad31e (stale) Mirror 20: 0xc40ad31e (stale) Mirror 21: 0xd922b1f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19d73: skipped 000975 1786940176.115344 punch from 0x01780d to 0x019d72 (0x2566 bytes) 000976 1786940176.117482 write 0x00771c thru 0x00b64f (0x3f34 bytes) 000977 1786940176.123307 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19d73): Mirror 20: 0xc40ad31e (stale) Mirror 21: 0xd922b1f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19d73): Mirror 20: 0xc40ad31e (stale) Mirror 21: 0xd922b1f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19d73: skipped 000978 1786940176.206510 trunc from 0x00647a to 0x013919 (0xd4a0 bytes) 000979 1786940176.208555 read 0x006b5f thru 0x00cbb9 (0x605b bytes) 000980 1786940176.211898 read 0x001c05 thru 0x0087ef (0x6beb bytes) 000981 1786940176.215137 punch from 0x00d693 to 0x017460 (0x9dce bytes) 000982 1786940176.217256 fallocate 0x0039f1 thru 0x011ab8 (0xe0c8 bytes) 000983 1786940176.219264 trunc from 0x010ed9 to 0x019d72 (0x8e9a bytes) 000984 1786940176.220909 mapread 0x001249 thru 0x0025bb (0x1373 bytes) 000985 1786940176.221841 punch from 0x01822b to 0x019d72 (0x1b48 bytes) 000986 1786940176.223619 mapread 0x0097a7 thru 0x016fbd (0xd817 bytes) 000987 1786940176.237498 write 0x02ba8e thru 0x035480 (0x99f3 bytes) 000988 1786940176.239598 read 0x024ad8 thru 0x032cf0 (0xe219 bytes) 000989 1786940176.242289 fallocate 0x00d05c thru 0x012e18 (0x5dbd bytes) 000990 1786940176.243979 mapwrite 0x015623 thru 0x02249b (0xce79 bytes) 000991 1786940176.252880 punch from 0x000c96 to 0x00a418 (0x9783 bytes) 000992 1786940176.255052 mapread 0x0294f3 thru 0x035480 (0xbf8e bytes) 000994 1786940176.256318 trunc from 0x02142f to 0x02ae04 (0x99d6 bytes) 000996 1786940176.259188 trunc from 0x0325af to 0x035480 (0x2ed2 bytes) 000997 1786940176.262329 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x805:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35481): Mirror 20: 0x9567b029 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35481): Mirror 20: 0x9567b029 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35481: skipped 000998 1786940176.355060 punch from 0x021f3f to 0x03161f (0xf6e1 bytes) 000999 1786940176.357226 read 0x00764c thru 0x00ada3 (0x3758 bytes) 001000 1786940176.359509 read 0x01d136 thru 0x023f1f (0x6dea bytes) All operations completed A-OK! PASS 70a (21s) == sanity-flr test 71: check mirror extend parent fid ==== 00:16:18 (1786940178) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (2s) == sanity-flr test 72: check append on FLR file ========== 00:16:21 (1786940181) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc66:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc82:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0557149 s, 37.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc66:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc67:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc82:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347997 s, 30.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc66:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc67:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc83:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc82:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc66:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc67:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc83:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc82:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc68:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 00:16:23 (1786940183) Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre2 Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6147040 .. Extending file size to 4811968 .. failed Extending file size to 8516992 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6284960 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7551936 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8877472 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8243840 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8363776 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8933216 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9015360 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8778400 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9064544 .. failed Extending file size to 9058304 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9204512 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9065184 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 65752 65753 65755 65757 65759 192.168.201.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre2 (opts:) 192.168.201.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 00:17:28 (1786940248) Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre2 Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8401088 .. Extending file size to 1213760 .. Extending file size to 4531616 .. done Extending file size to 8689600 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8742144 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8829152 .. Extending file size to 8777632 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9119040 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9136224 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8927520 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9013920 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9048896 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 69321 69322 69324 69326 69328 done 192.168.201.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre2 (opts:) 192.168.201.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 00:18:32 (1786940312) Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3748 1267556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18876 3588144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24656 3582364 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43532 7170508 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3748 1267556 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18876 3588144 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24656 3582364 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43532 7170508 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0343526 s, 61.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0482757 s, 43.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xddb54ca1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 00:18:38 (1786940318) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0297434 s, 70.5 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 00:18:40 (1786940320) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x850:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc99:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc99:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7d:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 00:18:42 (1786940322) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0246925 s, 84.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 00:18:45 (1786940325) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0175345 s, 120 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7f:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 00:18:47 (1786940327) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0243142 s, 86.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc80:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc80:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc81:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 00:19:11 (1786940351) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg133-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175251 s, 59.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc83:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc82:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc83:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc82:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9f:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:19:46 (1786940386) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg133-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0240828 s, 87.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:20:22 (1786940422) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg133-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg133-server: Pool lustre.test_204f created Adding targets to pool oleg133-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg133-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0241914 s, 86.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc84:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc84:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg133-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg133-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg133-server: Pool lustre.test_204f destroyed PASS 204f (47s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 00:21:09 (1786940469) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 00:21:11 (1786940471) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:21:14 (1786940474) oleg133-server: Pool lustre.test_206 created Adding targets to pool oleg133-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg133-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg133-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg133-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg133-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:21:29 (1786940489) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259854 s, 40.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132189 s, 79.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6e41a632 Mirror 3: 0x6e41a632 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:21:32 (1786940492) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:21:33 (1786940493) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:21:34 (1786940494) Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.00927932 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2610000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x13e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x13e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x13e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2180000 Read this (4 bytes): BBBB 192.168.201.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:21:50 (1786940510) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00352349 s, 2.3 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:21:52 (1786940512) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204637 s, 51.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x871 lmm_fid: [0x200000405:0x871:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x871 lmm_fid: [0x200000405:0x871:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcaa:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339613297 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-17 00:21:52.000000000 -0400 Modify: 2026-08-17 00:21:52.000000000 -0400 Change: 2026-08-17 00:21:53.000000000 -0400 Birth: 2026-08-17 00:21:52.000000000 -0400 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:21:54 (1786940514) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138869 s, 75.5 MB/s oleg133-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg133-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg133-server: O/280000401/d11/cab: File not found by ext2_lookup PASS 210b (12s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:22:06 (1786940526) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0433181 s, 946 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00463765 s, 883 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 17 00:22 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x876:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 17 00:22 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 00:22:08 (1786940528) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 00:22:11 (1786940531) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (2s) == sanity-flr test complete, duration 862 sec ============ 00:22:13 (1786940533) === sanity-flr: start cleanup 00:22:14 (1786940534) === === sanity-flr: finish cleanup 00:22:16 (1786940536) ===